IRIX Base Documentation 2002 November/SGI IRIX Base Documentation 2002 November.iso/usr/share/catman/p_man/cat1/flex.z/flex

Text File | 2002-10-03 | 179.5 KB | 3,697 lines

     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



     NNNNAAAAMMMMEEEE
          flex - fast lexical analyzer generator

     SSSSYYYYNNNNOOOOPPPPSSSSIIIISSSS
          fffflllleeeexxxx [[[[----bbbbccccddddffffhhhhiiiillllnnnnppppssssttttvvvvwwwwBBBBFFFFIIIILLLLTTTTVVVVXXXX77778888++++???? ----CCCC[[[[aaaaeeeeffffFFFFmmmmrrrr]]]] ----oooooooouuuuttttppppuuuutttt ----PPPPpppprrrreeeeffffiiiixxxx
          ----SSSSsssskkkkeeeelllleeeettttoooonnnn]]]] [[[[--------hhhheeeellllpppp --------vvvveeeerrrrssssiiiioooonnnn --------ppppoooossssiiiixxxx----ccccoooommmmppppaaaatttt]]]] [_f_i_l_e_n_a_m_e ...]

     OOOOVVVVEEEERRRRVVVVIIIIEEEEWWWW
          This manual describes _f_l_e_x, a tool for generating programs
          that perform pattern-matching on text.  The manual includes
          both tutorial and reference sections:

              Description
                  a brief overview of the tool

              Some Simple Examples

              Format Of The Input File

              Patterns
                  the extended regular expressions used by flex

              How The Input Is Matched
                  the rules for determining what has been matched

              Actions
                  how to specify what to do when a pattern is matched

              The Generated Scanner
                  details regarding the scanner that flex produces;
                  how to control the input source

              Start Conditions
                  introducing context into your scanners, and
                  managing "mini-scanners"

              Multiple Input Buffers
                  how to manipulate multiple input sources; how to
                  scan from strings instead of files

              End-of-file Rules
                  special rules for matching the end of the input

              Miscellaneous Macros
                  a summary of macros available to the actions

              Values Available To The User
                  a summary of values available to the actions

              Interfacing With Yacc
                  connecting flex scanners together with yacc parsers




     Page 1                                          (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



              Options
                  flex command-line options, and the "%option"
                  directive

              Performance Considerations
                  how to make your scanner go as fast as possible

              Generating C++ Scanners
                  the (experimental) facility for generating C++
                  scanner classes

              Incompatibilities With Lex And POSIX
                  how flex differs from AT&T lex and the POSIX lex
                  standard

              Diagnostics
                  those error messages produced by flex (or scanners
                  it generates) whose meanings might not be apparent

              Files
                  files used by flex

              Deficiencies / Bugs
                  known problems with flex

              See Also
                  other documentation, related tools

              Author
                  includes contact information


     DDDDEEEESSSSCCCCRRRRIIIIPPPPTTTTIIIIOOOONNNN
          _f_l_e_x is a tool for generating _s_c_a_n_n_e_r_s: programs which
          recognized lexical patterns in text.  _f_l_e_x reads the given
          input files, or its standard input if no file names are
          given, for a description of a scanner to generate.  The
          description is in the form of pairs of regular expressions
          and C code, called _r_u_l_e_s. _f_l_e_x generates as output a C
          source file, lllleeeexxxx....yyyyyyyy....cccc,,,, which defines a routine yyyyyyyylllleeeexxxx(((()))).... This
          file is compiled and linked with the ----llllffffllll library to produce
          an executable.  When the executable is run, it analyzes its
          input for occurrences of the regular expressions.  Whenever
          it finds one, it executes the corresponding C code.

     SSSSOOOOMMMMEEEE SSSSIIIIMMMMPPPPLLLLEEEE EEEEXXXXAAAAMMMMPPPPLLLLEEEESSSS
          First some simple examples to get the flavor of how one uses
          _f_l_e_x. The following _f_l_e_x input specifies a scanner which
          whenever it encounters the string "username" will replace it
          with the user's login name:

              %%



     Page 2                                          (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



              username    printf( "%s", getlogin() );

          By default, any text not matched by a _f_l_e_x scanner is copied
          to the output, so the net effect of this scanner is to copy
          its input file to its output with each occurrence of
          "username" expanded.  In this input, there is just one rule.
          "username" is the _p_a_t_t_e_r_n and the "printf" is the _a_c_t_i_o_n.
          The "%%" marks the beginning of the rules.

          Here's another simple example:

                      int num_lines = 0, num_chars = 0;

              %%
              \n      ++num_lines; ++num_chars;
              .       ++num_chars;

              %%
              main()
                      {
                      yylex();
                      printf( "# of lines = %d, # of chars = %d\n",
                              num_lines, num_chars );
                      }

          This scanner counts the number of characters and the number
          of lines in its input (it produces no output other than the
          final report on the counts).  The first line declares two
          globals, "num_lines" and "num_chars", which are accessible
          both inside yyyyyyyylllleeeexxxx(((()))) and in the mmmmaaaaiiiinnnn(((()))) routine declared after
          the second "%%".  There are two rules, one which matches a
          newline ("\n") and increments both the line count and the
          character count, and one which matches any character other
          than a newline (indicated by the "." regular expression).

          A somewhat more complicated example:

              /* scanner for a toy Pascal-like language */

              %{
              /* need this for the call to atof() below */
              #include <math.h>
              %}

              DIGIT    [0-9]
              ID       [a-z][a-z0-9]*

              %%

              {DIGIT}+    {
                          printf( "An integer: %s (%d)\n", yytext,
                                  atoi( yytext ) );



     Page 3                                          (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



                          }

              {DIGIT}+"."{DIGIT}*        {
                          printf( "A float: %s (%g)\n", yytext,
                                  atof( yytext ) );
                          }

              if|then|begin|end|procedure|function        {
                          printf( "A keyword: %s\n", yytext );
                          }

              {ID}        printf( "An identifier: %s\n", yytext );

              "+"|"-"|"*"|"/"   printf( "An operator: %s\n", yytext );

              "{"[^}\n]*"}"     /* eat up one-line comments */

              [ \t\n]+          /* eat up whitespace */

              .           printf( "Unrecognized character: %s\n", yytext );

              %%

              main( argc, argv )
              int argc;
              char **argv;
                  {
                  ++argv, --argc;  /* skip over program name */
                  if ( argc > 0 )
                          yyin = fopen( argv[0], "r" );
                  else
                          yyin = stdin;

                  yylex();
                  }

          This is the beginnings of a simple scanner for a language
          like Pascal.  It identifies different types of _t_o_k_e_n_s and
          reports on what it has seen.

          The details of this example will be explained in the
          following sections.

     FFFFOOOORRRRMMMMAAAATTTT OOOOFFFF TTTTHHHHEEEE IIIINNNNPPPPUUUUTTTT FFFFIIIILLLLEEEE
          The _f_l_e_x input file consists of three sections, separated by
          a line with just %%%%%%%% in it:

              definitions
              %%
              rules
              %%
              user code



     Page 4                                          (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          The _d_e_f_i_n_i_t_i_o_n_s section contains declarations of simple _n_a_m_e
          definitions to simplify the scanner specification, and
          declarations of _s_t_a_r_t _c_o_n_d_i_t_i_o_n_s, which are explained in a
          later section.

          Name definitions have the form:

              name definition

          The "name" is a word beginning with a letter or an
          underscore ('_') followed by zero or more letters, digits,
          '_', or '-' (dash).  The definition is taken to begin at the
          first non-white-space character following the name and
          continuing to the end of the line.  The definition can
          subsequently be referred to using "{name}", which will
          expand to "(definition)".  For example,

              DIGIT    [0-9]
              ID       [a-z][a-z0-9]*

          defines "DIGIT" to be a regular expression which matches a
          single digit, and "ID" to be a regular expression which
          matches a letter followed by zero-or-more letters-or-digits.
          A subsequent reference to

              {DIGIT}+"."{DIGIT}*

          is identical to

              ([0-9])+"."([0-9])*

          and matches one-or-more digits followed by a '.' followed by
          zero-or-more digits.

          The _r_u_l_e_s section of the _f_l_e_x input contains a series of
          rules of the form:

              pattern   action

          where the pattern must be unindented and the action must
          begin on the same line.

          See below for a further description of patterns and actions.

          Finally, the user code section is simply copied to lllleeeexxxx....yyyyyyyy....cccc
          verbatim.  It is used for companion routines which call or
          are called by the scanner.  The presence of this section is
          optional; if it is missing, the second %%%%%%%% in the input file
          may be skipped, too.

          In the definitions and rules sections, any _i_n_d_e_n_t_e_d text or
          text enclosed in %%%%{{{{ and %%%%}}}} is copied verbatim to the output



     Page 5                                          (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          (with the %{}'s removed).  The %{}'s must appear unindented
          on lines by themselves.

          In the rules section, any indented or %{} text appearing
          before the first rule may be used to declare variables which
          are local to the scanning routine and (after the
          declarations) code which is to be executed whenever the
          scanning routine is entered.  Other indented or %{} text in
          the rule section is still copied to the output, but its
          meaning is not well-defined and it may well cause compile-
          time errors (this feature is present for _P_O_S_I_X compliance;
          see below for other such features).

          In the definitions section (but not in the rules section),
          an unindented comment (i.e., a line beginning with "/*") is
          also copied verbatim to the output up to the next "*/".

     PPPPAAAATTTTTTTTEEEERRRRNNNNSSSS
          The patterns in the input are written using an extended set
          of regular expressions.  These are:

              x          match the character 'x'
              .          any character (byte) except newline
              [xyz]      a "character class"; in this case, the pattern
                           matches either an 'x', a 'y', or a 'z'
              [abj-oZ]   a "character class" with a range in it; matches
                           an 'a', a 'b', any letter from 'j' through 'o',
                           or a 'Z'
              [^A-Z]     a "negated character class", i.e., any character
                           but those in the class.  In this case, any
                           character EXCEPT an uppercase letter.
              [^A-Z\n]   any character EXCEPT an uppercase letter or
                           a newline
              r*         zero or more r's, where r is any regular expression
              r+         one or more r's
              r?         zero or one r's (that is, "an optional r")
              r{2,5}     anywhere from two to five r's
              r{2,}      two or more r's
              r{4}       exactly 4 r's
              {name}     the expansion of the "name" definition
                         (see above)
              "[xyz]\"foo"
                         the literal string: [xyz]"foo
              \X         if X is an 'a', 'b', 'f', 'n', 'r', 't', or 'v',
                           then the ANSI-C interpretation of \x.
                           Otherwise, a literal 'X' (used to escape
                           operators such as '*')
              \0         a NUL character (ASCII code 0)
              \123       the character with octal value 123
              \x2a       the character with hexadecimal value 2a
              (r)        match an r; parentheses are used to override
                           precedence (see below)



     Page 6                                          (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



              rs         the regular expression r followed by the
                           regular expression s; called "concatenation"


              r|s        either an r or an s


              r/s        an r but only if it is followed by an s.  The
                           text matched by s is included when determining
                           whether this rule is the "longest match",
                           but is then returned to the input before
                           the action is executed.  So the action only
                           sees the text matched by r.  This type
                           of pattern is called trailing context".
                           (There are some combinations of r/s that flex
                           cannot match correctly; see notes in the
                           Deficiencies / Bugs section below regarding
                           "dangerous trailing context".)
              ^r         an r, but only at the beginning of a line (i.e.,
                           which just starting to scan, or right after a
                           newline has been scanned).
              r$         an r, but only at the end of a line (i.e., just
                           before a newline).  Equivalent to "r/\n".

                         Note that flex's notion of "newline" is exactly
                         whatever the C compiler used to compile flex
                         interprets '\n' as; in particular, on some DOS
                         systems you must either filter out \r's in the
                         input yourself, or explicitly use r/\r\n for "r$".


              <s>r       an r, but only in start condition s (see
                           below for discussion of start conditions)
              <s1,s2,s3>r
                         same, but in any of start conditions s1,
                           s2, or s3
              <*>r       an r in any start condition, even an exclusive one.


              <<EOF>>    an end-of-file
              <s1,s2><<EOF>>
                         an end-of-file when in start condition s1 or s2

          Note that inside of a character class, all regular
          expression operators lose their special meaning except
          escape ('\') and the character class operators, '-', ']',
          and, at the beginning of the class, '^'.

          The regular expressions listed above are grouped according
          to precedence, from highest precedence at the top to lowest
          at the bottom.  Those grouped together have equal precedence
          (see special note on the precedence of the repeat operator,



     Page 7                                          (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          {{{{}}}}, under the documentation for the --------ppppoooossssiiiixxxx----ccccoooommmmppppaaaatttt POSIX
          compliance option).  For example,

              foo|bar*

          is the same as

              (foo)|(ba(r*))

          since the '*' operator has higher precedence than
          concatenation, and concatenation higher than alternation
          ('|').  This pattern therefore matches _e_i_t_h_e_r the string
          "foo" _o_r the string "ba" followed by zero-or-more r's.  To
          match "foo" or zero-or-more "bar"'s, use:

              foo|(bar)*

          and to match zero-or-more "foo"'s-or-"bar"'s:

              (foo|bar)*


          In addition to characters and ranges of characters,
          character classes can also contain character class
          _e_x_p_r_e_s_s_i_o_n_s. These are expressions enclosed inside [[[[:::: and ::::]]]]
          delimiters (which themselves must appear between the '[' and
          ']' of the character class; other elements may occur inside
          the character class, too).  The valid expressions are:

              [:alnum:] [:alpha:] [:blank:]
              [:cntrl:] [:digit:] [:graph:]
              [:lower:] [:print:] [:punct:]
              [:space:] [:upper:] [:xdigit:]

          These expressions all designate a set of characters
          equivalent to the corresponding standard C iiiissssXXXXXXXXXXXX function.
          For example, [[[[::::aaaallllnnnnuuuummmm::::]]]] designates those characters for which
          iiiissssaaaallllnnnnuuuummmm(((()))) returns true - i.e., any alphabetic or numeric.
          Some systems don't provide iiiissssbbbbllllaaaannnnkkkk(((()))),,,, so flex defines
          [[[[::::bbbbllllaaaannnnkkkk::::]]]] as a blank or a tab.

          For example, the following character classes are all
          equivalent:

              [[:alnum:]]
              [[:alpha:][:digit:]
              [[:alpha:]0-9]
              [a-zA-Z0-9]

          If your scanner is case-insensitive (the ----iiii flag), then
          [[[[::::uuuuppppppppeeeerrrr::::]]]] and [[[[::::lllloooowwwweeeerrrr::::]]]] are equivalent to [[[[::::aaaallllpppphhhhaaaa::::]]]]....




     Page 8                                          (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          Some notes on patterns:

          -    A negated character class such as the example "[^A-Z]"
               above _w_i_l_l _m_a_t_c_h _a _n_e_w_l_i_n_e unless "\n" (or an
               equivalent escape sequence) is one of the characters
               explicitly present in the negated character class
               (e.g., "[^A-Z\n]").  This is unlike how many other
               regular expression tools treat negated character
               classes, but unfortunately the inconsistency is
               historically entrenched.  Matching newlines means that
               a pattern like [^"]* can match the entire input unless
               there's another quote in the input.

          -    A rule can have at most one instance of trailing
               context (the '/' operator or the '$' operator).  The
               start condition, '^', and "<<EOF>>" patterns can only
               occur at the beginning of a pattern, and, as well as
               with '/' and '$', cannot be grouped inside parentheses.
               A '^' which does not occur at the beginning of a rule
               or a '$' which does not occur at the end of a rule
               loses its special properties and is treated as a normal
               character.

               The following are illegal:

                   foo/bar$
                   <sc1>foo<sc2>bar

               Note that the first of these, can be written
               "foo/bar\n".

               The following will result in '$' or '^' being treated
               as a normal character:

                   foo|(bar$)
                   foo|^bar

               If what's wanted is a "foo" or a bar-followed-by-a-
               newline, the following could be used (the special '|'
               action is explained below):

                   foo      |
                   bar$     /* action goes here */

               A similar trick will work for matching a foo or a bar-
               at-the-beginning-of-a-line.

     HHHHOOOOWWWW TTTTHHHHEEEE IIIINNNNPPPPUUUUTTTT IIIISSSS MMMMAAAATTTTCCCCHHHHEEEEDDDD
          When the generated scanner is run, it analyzes its input
          looking for strings which match any of its patterns.  If it
          finds more than one match, it takes the one matching the
          most text (for trailing context rules, this includes the



     Page 9                                          (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          length of the trailing part, even though it will then be
          returned to the input).  If it finds two or more matches of
          the same length, the rule listed first in the _f_l_e_x input
          file is chosen.

          Once the match is determined, the text corresponding to the
          match (called the _t_o_k_e_n) is made available in the global
          character pointer yyyyyyyytttteeeexxxxtttt,,,, and its length in the global
          integer yyyyyyyylllleeeennnngggg.... The _a_c_t_i_o_n corresponding to the matched
          pattern is then executed (a more detailed description of
          actions follows), and then the remaining input is scanned
          for another match.

          If no match is found, then the _d_e_f_a_u_l_t _r_u_l_e is executed: the
          next character in the input is considered matched and copied
          to the standard output.  Thus, the simplest legal _f_l_e_x input
          is:

              %%

          which generates a scanner that simply copies its input (one
          character at a time) to its output.

          Note that yyyyyyyytttteeeexxxxtttt can be defined in two different ways:
          either as a character _p_o_i_n_t_e_r or as a character _a_r_r_a_y. You
          can control which definition _f_l_e_x uses by including one of
          the special directives %%%%ppppooooiiiinnnntttteeeerrrr or %%%%aaaarrrrrrrraaaayyyy in the first
          (definitions) section of your flex input.  The default is
          %%%%ppppooooiiiinnnntttteeeerrrr,,,, unless you use the ----llll lex compatibility option, in
          which case yyyyyyyytttteeeexxxxtttt will be an array.  The advantage of using
          %%%%ppppooooiiiinnnntttteeeerrrr is substantially faster scanning and no buffer
          overflow when matching very large tokens (unless you run out
          of dynamic memory).  The disadvantage is that you are
          restricted in how your actions can modify yyyyyyyytttteeeexxxxtttt (see the
          next section), and calls to the uuuunnnnppppuuuutttt(((()))) function destroys
          the present contents of yyyyyyyytttteeeexxxxtttt,,,, which can be a considerable
          porting headache when moving between different _l_e_x versions.

          The advantage of %%%%aaaarrrrrrrraaaayyyy is that you can then modify yyyyyyyytttteeeexxxxtttt
          to your heart's content, and calls to uuuunnnnppppuuuutttt(((()))) do not destroy
          yyyyyyyytttteeeexxxxtttt (see below).  Furthermore, existing _l_e_x programs
          sometimes access yyyyyyyytttteeeexxxxtttt externally using declarations of the
          form:
              extern char yytext[];
          This definition is erroneous when used with %%%%ppppooooiiiinnnntttteeeerrrr,,,, but
          correct for %%%%aaaarrrrrrrraaaayyyy....

          %%%%aaaarrrrrrrraaaayyyy defines yyyyyyyytttteeeexxxxtttt to be an array of YYYYYYYYLLLLMMMMAAAAXXXX characters,
          which defaults to a fairly large value.  You can change the
          size by simply #define'ing YYYYYYYYLLLLMMMMAAAAXXXX to a different value in
          the first section of your _f_l_e_x input.  As mentioned above,
          with %%%%ppppooooiiiinnnntttteeeerrrr yytext grows dynamically to accommodate large



     Page 10                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          tokens.  While this means your %%%%ppppooooiiiinnnntttteeeerrrr scanner can
          accommodate very large tokens (such as matching entire
          blocks of comments), bear in mind that each time the scanner
          must resize yyyyyyyytttteeeexxxxtttt it also must rescan the entire token from
          the beginning, so matching such tokens can prove slow.
          yyyyyyyytttteeeexxxxtttt presently does _n_o_t dynamically grow if a call to
          uuuunnnnppppuuuutttt(((()))) results in too much text being pushed back; instead,
          a run-time error results.

          Also note that you cannot use %%%%aaaarrrrrrrraaaayyyy with C++ scanner
          classes (the cccc++++++++ option; see below).

     AAAACCCCTTTTIIIIOOOONNNNSSSS
          Each pattern in a rule has a corresponding action, which can
          be any arbitrary C statement.  The pattern ends at the first
          non-escaped whitespace character; the remainder of the line
          is its action.  If the action is empty, then when the
          pattern is matched the input token is simply discarded.  For
          example, here is the specification for a program which
          deletes all occurrences of "zap me" from its input:

              %%
              "zap me"

          (It will copy all other characters in the input to the
          output since they will be matched by the default rule.)

          Here is a program which compresses multiple blanks and tabs
          down to a single blank, and throws away whitespace found at
          the end of a line:

              %%
              [ \t]+        putchar( ' ' );
              [ \t]+$       /* ignore this token */


          If the action contains a '{', then the action spans till the
          balancing '}' is found, and the action may cross multiple
          lines.  _f_l_e_x knows about C strings and comments and won't be
          fooled by braces found within them, but also allows actions
          to begin with %%%%{{{{ and will consider the action to be all the
          text up to the next %%%%}}}} (regardless of ordinary braces inside
          the action).

          An action consisting solely of a vertical bar ('|') means
          "same as the action for the next rule."  See below for an
          illustration.

          Actions can include arbitrary C code, including rrrreeeettttuuuurrrrnnnn
          statements to return a value to whatever routine called
          yyyyyyyylllleeeexxxx(((()))).... Each time yyyyyyyylllleeeexxxx(((()))) is called it continues processing
          tokens from where it last left off until it either reaches



     Page 11                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          the end of the file or executes a return.

          Actions are free to modify yyyyyyyytttteeeexxxxtttt except for lengthening it
          (adding characters to its end--these will overwrite later
          characters in the input stream).  This however does not
          apply when using %%%%aaaarrrrrrrraaaayyyy (see above); in that case, yyyyyyyytttteeeexxxxtttt
          may be freely modified in any way.

          Actions are free to modify yyyyyyyylllleeeennnngggg except they should not do
          so if the action also includes use of yyyyyyyymmmmoooorrrreeee(((()))) (see below).

          There are a number of special directives which can be
          included within an action:

          -    EEEECCCCHHHHOOOO copies yytext to the scanner's output.

          -    BBBBEEEEGGGGIIIINNNN followed by the name of a start condition places
               the scanner in the corresponding start condition (see
               below).

          -    RRRREEEEJJJJEEEECCCCTTTT directs the scanner to proceed on to the "second
               best" rule which matched the input (or a prefix of the
               input).  The rule is chosen as described above in "How
               the Input is Matched", and yyyyyyyytttteeeexxxxtttt and yyyyyyyylllleeeennnngggg set up
               appropriately.  It may either be one which matched as
               much text as the originally chosen rule but came later
               in the _f_l_e_x input file, or one which matched less text.
               For example, the following will both count the words in
               the input and call the routine special() whenever
               "frob" is seen:

                           int word_count = 0;
                   %%

                   frob        special(); REJECT;
                   [^ \t\n]+   ++word_count;

               Without the RRRREEEEJJJJEEEECCCCTTTT,,,, any "frob"'s in the input would not
               be counted as words, since the scanner normally
               executes only one action per token.  Multiple RRRREEEEJJJJEEEECCCCTTTT''''ssss
               are allowed, each one finding the next best choice to
               the currently active rule.  For example, when the
               following scanner scans the token "abcd", it will write
               "abcdabcaba" to the output:

                   %%
                   a        |
                   ab       |
                   abc      |
                   abcd     ECHO; REJECT;
                   .|\n     /* eat up any unmatched character */




     Page 12                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



               (The first three rules share the fourth's action since
               they use the special '|' action.)  RRRREEEEJJJJEEEECCCCTTTT is a
               particularly expensive feature in terms of scanner
               performance; if it is used in _a_n_y of the scanner's
               actions it will slow down _a_l_l of the scanner's
               matching.  Furthermore, RRRREEEEJJJJEEEECCCCTTTT cannot be used with the
               -_C_f or -_C_F options (see below).

               Note also that unlike the other special actions, RRRREEEEJJJJEEEECCCCTTTT
               is a _b_r_a_n_c_h; code immediately following it in the
               action will _n_o_t be executed.

          -    yyyyyyyymmmmoooorrrreeee(((()))) tells the scanner that the next time it
               matches a rule, the corresponding token should be
               _a_p_p_e_n_d_e_d onto the current value of yyyyyyyytttteeeexxxxtttt rather than
               replacing it.  For example, given the input "mega-
               kludge" the following will write "mega-mega-kludge" to
               the output:

                   %%
                   mega-    ECHO; yymore();
                   kludge   ECHO;

               First "mega-" is matched and echoed to the output.
               Then "kludge" is matched, but the previous "mega-" is
               still hanging around at the beginning of yyyyyyyytttteeeexxxxtttt so the
               EEEECCCCHHHHOOOO for the "kludge" rule will actually write "mega-
               kludge".

          Two notes regarding use of yyyyyyyymmmmoooorrrreeee(((()))).... First, yyyyyyyymmmmoooorrrreeee(((()))) depends
          on the value of _y_y_l_e_n_g correctly reflecting the size of the
          current token, so you must not modify _y_y_l_e_n_g if you are
          using yyyyyyyymmmmoooorrrreeee(((()))).... Second, the presence of yyyyyyyymmmmoooorrrreeee(((()))) in the
          scanner's action entails a minor performance penalty in the
          scanner's matching speed.

          -    yyyyyyyylllleeeessssssss((((nnnn)))) returns all but the first _n characters of the
               current token back to the input stream, where they will
               be rescanned when the scanner looks for the next match.
               yyyyyyyytttteeeexxxxtttt and yyyyyyyylllleeeennnngggg are adjusted appropriately (e.g.,
               yyyyyyyylllleeeennnngggg will now be equal to _n ).  For example, on the
               input "foobar" the following will write out
               "foobarbar":

                   %%
                   foobar    ECHO; yyless(3);
                   [a-z]+    ECHO;

               An argument of 0 to yyyyyyyylllleeeessssssss will cause the entire
               current input string to be scanned again.  Unless
               you've changed how the scanner will subsequently
               process its input (using BBBBEEEEGGGGIIIINNNN,,,, for example), this will



     Page 13                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



               result in an endless loop.

          Note that yyyyyyyylllleeeessssssss is a macro and can only be used in the flex
          input file, not from other source files.

          -    uuuunnnnppppuuuutttt((((cccc)))) puts the character _c back onto the input
               stream.  It will be the next character scanned.  The
               following action will take the current token and cause
               it to be rescanned enclosed in parentheses.

                   {
                   int i;
                   /* Copy yytext because unput() trashes yytext */
                   char *yycopy = strdup( yytext );
                   unput( ')' );
                   for ( i = yyleng - 1; i >= 0; --i )
                       unput( yycopy[i] );
                   unput( '(' );
                   free( yycopy );
                   }

               Note that since each uuuunnnnppppuuuutttt(((()))) puts the given character
               back at the _b_e_g_i_n_n_i_n_g of the input stream, pushing back
               strings must be done back-to-front.

          An important potential problem when using uuuunnnnppppuuuutttt(((()))) is that if
          you are using %%%%ppppooooiiiinnnntttteeeerrrr (the default), a call to uuuunnnnppppuuuutttt(((())))
          _d_e_s_t_r_o_y_s the contents of _y_y_t_e_x_t, starting with its rightmost
          character and devouring one character to the left with each
          call.  If you need the value of yytext preserved after a
          call to uuuunnnnppppuuuutttt(((()))) (as in the above example), you must either
          first copy it elsewhere, or build your scanner using %%%%aaaarrrrrrrraaaayyyy
          instead (see How The Input Is Matched).

          Finally, note that you cannot put back EEEEOOOOFFFF to attempt to
          mark the input stream with an end-of-file.

          -    iiiinnnnppppuuuutttt(((()))) reads the next character from the input stream.
               For example, the following is one way to eat up C
               comments:

                   %%
                   "/*"        {
                               register int c;

                               for ( ; ; )
                                   {
                                   while ( (c = input()) != '*' &&
                                           c != EOF )
                                       ;    /* eat up text of comment */

                                   if ( c == '*' )



     Page 14                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



                                       {
                                       while ( (c = input()) == '*' )
                                           ;
                                       if ( c == '/' )
                                           break;    /* found the end */
                                       }

                                   if ( c == EOF )
                                       {
                                       error( "EOF in comment" );
                                       break;
                                       }
                                   }
                               }

               (Note that if the scanner is compiled using CCCC++++++++,,,, then
               iiiinnnnppppuuuutttt(((()))) is instead referred to as yyyyyyyyiiiinnnnppppuuuutttt(((()))),,,, in order
               to avoid a name clash with the CCCC++++++++ stream by the name
               of _i_n_p_u_t.)

          -    YYYYYYYY____FFFFLLLLUUUUSSSSHHHH____BBBBUUUUFFFFFFFFEEEERRRR flushes the scanner's internal buffer
               so that the next time the scanner attempts to match a
               token, it will first refill the buffer using YYYYYYYY____IIIINNNNPPPPUUUUTTTT
               (see The Generated Scanner, below).  This action is a
               special case of the more general yyyyyyyy____fffflllluuuusssshhhh____bbbbuuuuffffffffeeeerrrr(((())))
               function, described below in the section Multiple Input
               Buffers.

          -    yyyyyyyytttteeeerrrrmmmmiiiinnnnaaaatttteeee(((()))) can be used in lieu of a return statement
               in an action.  It terminates the scanner and returns a
               0 to the scanner's caller, indicating "all done".  By
               default, yyyyyyyytttteeeerrrrmmmmiiiinnnnaaaatttteeee(((()))) is also called when an end-of-
               file is encountered.  It is a macro and may be
               redefined.

     TTTTHHHHEEEE GGGGEEEENNNNEEEERRRRAAAATTTTEEEEDDDD SSSSCCCCAAAANNNNNNNNEEEERRRR
          The output of _f_l_e_x is the file lllleeeexxxx....yyyyyyyy....cccc,,,, which contains the
          scanning routine yyyyyyyylllleeeexxxx(((()))),,,, a number of tables used by it for
          matching tokens, and a number of auxiliary routines and
          macros.  By default, yyyyyyyylllleeeexxxx(((()))) is declared as follows:

              int yylex()
                  {
                  ... various definitions and the actions in here ...
                  }

          (If your environment supports function prototypes, then it
          will be "int yylex( void )".)  This definition may be
          changed by defining the "YY_DECL" macro.  For example, you
          could use:

              #define YY_DECL float lexscan( a, b ) float a, b;



     Page 15                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          to give the scanning routine the name _l_e_x_s_c_a_n, returning a
          float, and taking two floats as arguments.  Note that if you
          give arguments to the scanning routine using a K&R-
          style/non-prototyped function declaration, you must
          terminate the definition with a semi-colon (;).

          Whenever yyyyyyyylllleeeexxxx(((()))) is called, it scans tokens from the global
          input file _y_y_i_n (which defaults to stdin).  It continues
          until it either reaches an end-of-file (at which point it
          returns the value 0) or one of its actions executes a _r_e_t_u_r_n
          statement.

          If the scanner reaches an end-of-file, subsequent calls are
          undefined unless either _y_y_i_n is pointed at a new input file
          (in which case scanning continues from that file), or
          yyyyyyyyrrrreeeessssttttaaaarrrrtttt(((()))) is called.  yyyyyyyyrrrreeeessssttttaaaarrrrtttt(((()))) takes one argument, a
          FFFFIIIILLLLEEEE **** pointer (which can be nil, if you've set up YYYYYYYY____IIIINNNNPPPPUUUUTTTT
          to scan from a source other than _y_y_i_n), and initializes _y_y_i_n
          for scanning from that file.  Essentially there is no
          difference between just assigning _y_y_i_n to a new input file
          or using yyyyyyyyrrrreeeessssttttaaaarrrrtttt(((()))) to do so; the latter is available for
          compatibility with previous versions of _f_l_e_x, and because it
          can be used to switch input files in the middle of scanning.
          It can also be used to throw away the current input buffer,
          by calling it with an argument of _y_y_i_n; but better is to use
          YYYYYYYY____FFFFLLLLUUUUSSSSHHHH____BBBBUUUUFFFFFFFFEEEERRRR (see above).  Note that yyyyyyyyrrrreeeessssttttaaaarrrrtttt(((()))) does _n_o_t
          reset the start condition to IIIINNNNIIIITTTTIIIIAAAALLLL (see Start Conditions,
          below).

          If yyyyyyyylllleeeexxxx(((()))) stops scanning due to executing a _r_e_t_u_r_n
          statement in one of the actions, the scanner may then be
          called again and it will resume scanning where it left off.

          By default (and for purposes of efficiency), the scanner
          uses block-reads rather than simple _g_e_t_c() calls to read
          characters from _y_y_i_n. The nature of how it gets its input
          can be controlled by defining the YYYYYYYY____IIIINNNNPPPPUUUUTTTT macro.
          YY_INPUT's calling sequence is
          "YY_INPUT(buf,result,max_size)".  Its action is to place up
          to _m_a_x__s_i_z_e characters in the character array _b_u_f and return
          in the integer variable _r_e_s_u_l_t either the number of
          characters read or the constant YY_NULL (0 on Unix systems)
          to indicate EOF.  The default YY_INPUT reads from the global
          file-pointer "yyin".

          A sample definition of YY_INPUT (in the definitions section
          of the input file):

              %{
              #define YY_INPUT(buf,result,max_size) \
                  { \
                  int c = getchar(); \



     Page 16                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



                  result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \
                  }
              %}

          This definition will change the input processing to occur
          one character at a time.

          When the scanner receives an end-of-file indication from
          YY_INPUT, it then checks the yyyyyyyywwwwrrrraaaapppp(((()))) function.  If yyyyyyyywwwwrrrraaaapppp(((())))
          returns false (zero), then it is assumed that the function
          has gone ahead and set up _y_y_i_n to point to another input
          file, and scanning continues.  If it returns true (non-
          zero), then the scanner terminates, returning 0 to its
          caller.  Note that in either case, the start condition
          remains unchanged; it does _n_o_t revert to IIIINNNNIIIITTTTIIIIAAAALLLL....

          If you do not supply your own version of yyyyyyyywwwwrrrraaaapppp(((()))),,,, then you
          must either use %%%%ooooppppttttiiiioooonnnn nnnnooooyyyyyyyywwwwrrrraaaapppp (in which case the scanner
          behaves as though yyyyyyyywwwwrrrraaaapppp(((()))) returned 1), or you must link
          with ----llllffffllll to obtain the default version of the routine,
          which always returns 1.

          Three routines are available for scanning from in-memory
          buffers rather than files:  yyyyyyyy____ssssccccaaaannnn____ssssttttrrrriiiinnnngggg(((()))),,,,
          yyyyyyyy____ssssccccaaaannnn____bbbbyyyytttteeeessss(((()))),,,, and yyyyyyyy____ssssccccaaaannnn____bbbbuuuuffffffffeeeerrrr(((()))).... See the discussion of
          them below in the section Multiple Input Buffers.

          The scanner writes its EEEECCCCHHHHOOOO output to the _y_y_o_u_t global
          (default, stdout), which may be redefined by the user simply
          by assigning it to some other FFFFIIIILLLLEEEE pointer.

     SSSSTTTTAAAARRRRTTTT CCCCOOOONNNNDDDDIIIITTTTIIIIOOOONNNNSSSS
          _f_l_e_x provides a mechanism for conditionally activating
          rules.  Any rule whose pattern is prefixed with "<sc>" will
          only be active when the scanner is in the start condition
          named "sc".  For example,

              <STRING>[^"]*        { /* eat up the string body ... */
                          ...
                          }

          will be active only when the scanner is in the "STRING"
          start condition, and

              <INITIAL,STRING,QUOTE>\.        { /* handle an escape ... */
                          ...
                          }

          will be active only when the current start condition is
          either "INITIAL", "STRING", or "QUOTE".

          Start conditions are declared in the definitions (first)



     Page 17                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          section of the input using unindented lines beginning with
          either %%%%ssss or %%%%xxxx followed by a list of names.  The former
          declares _i_n_c_l_u_s_i_v_e start conditions, the latter _e_x_c_l_u_s_i_v_e
          start conditions.  A start condition is activated using the
          BBBBEEEEGGGGIIIINNNN action.  Until the next BBBBEEEEGGGGIIIINNNN action is executed,
          rules with the given start condition will be active and
          rules with other start conditions will be inactive.  If the
          start condition is _i_n_c_l_u_s_i_v_e, then rules with no start
          conditions at all will also be active.  If it is _e_x_c_l_u_s_i_v_e,
          then _o_n_l_y rules qualified with the start condition will be
          active.  A set of rules contingent on the same exclusive
          start condition describe a scanner which is independent of
          any of the other rules in the _f_l_e_x input.  Because of this,
          exclusive start conditions make it easy to specify "mini-
          scanners" which scan portions of the input that are
          syntactically different from the rest (e.g., comments).

          If the distinction between inclusive and exclusive start
          conditions is still a little vague, here's a simple example
          illustrating the connection between the two.  The set of
          rules:

              %s example
              %%

              <example>foo   do_something();

              bar            something_else();

          is equivalent to

              %x example
              %%

              <example>foo   do_something();

              <INITIAL,example>bar    something_else();

          Without the <<<<IIIINNNNIIIITTTTIIIIAAAALLLL,,,,eeeexxxxaaaammmmpppplllleeee>>>> qualifier, the _b_a_r pattern in
          the second example wouldn't be active (i.e., couldn't match)
          when in start condition eeeexxxxaaaammmmpppplllleeee.... If we just used <<<<eeeexxxxaaaammmmpppplllleeee>>>>
          to qualify _b_a_r, though, then it would only be active in
          eeeexxxxaaaammmmpppplllleeee and not in IIIINNNNIIIITTTTIIIIAAAALLLL,,,, while in the first example it's
          active in both, because in the first example the eeeexxxxaaaammmmpppplllleeee
          startion condition is an _i_n_c_l_u_s_i_v_e ((((%%%%ssss)))) start condition.

          Also note that the special start-condition specifier <<<<****>>>>
          matches every start condition.  Thus, the above example
          could also have been written;

              %x example
              %%



     Page 18                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



              <example>foo   do_something();

              <*>bar    something_else();


          The default rule (to EEEECCCCHHHHOOOO any unmatched character) remains
          active in start conditions.  It is equivalent to:

              <*>.|\n     ECHO;


          BBBBEEEEGGGGIIIINNNN((((0000)))) returns to the original state where only the rules
          with no start conditions are active.  This state can also be
          referred to as the start-condition "INITIAL", so
          BBBBEEEEGGGGIIIINNNN((((IIIINNNNIIIITTTTIIIIAAAALLLL)))) is equivalent to BBBBEEEEGGGGIIIINNNN((((0000)))).... (The parentheses
          around the start condition name are not required but are
          considered good style.)

          BBBBEEEEGGGGIIIINNNN actions can also be given as indented code at the
          beginning of the rules section.  For example, the following
          will cause the scanner to enter the "SPECIAL" start
          condition whenever yyyyyyyylllleeeexxxx(((()))) is called and the global variable
          _e_n_t_e_r__s_p_e_c_i_a_l is true:

                      int enter_special;

              %x SPECIAL
              %%
                      if ( enter_special )
                          BEGIN(SPECIAL);

              <SPECIAL>blahblahblah
              ...more rules follow...


          To illustrate the uses of start conditions, here is a
          scanner which provides two different interpretations of a
          string like "123.456".  By default it will treat it as three
          tokens, the integer "123", a dot ('.'), and the integer
          "456".  But if the string is preceded earlier in the line by
          the string "expect-floats" it will treat it as a single
          token, the floating-point number 123.456:

              %{
              #include <math.h>
              %}
              %s expect

              %%
              expect-floats        BEGIN(expect);

              <expect>[0-9]+"."[0-9]+      {



     Page 19                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



                          printf( "found a float, = %f\n",
                                  atof( yytext ) );
                          }
              <expect>\n           {
                          /* that's the end of the line, so
                           * we need another "expect-number"
                           * before we'll recognize any more
                           * numbers
                           */
                          BEGIN(INITIAL);
                          }

              [0-9]+      {
                          printf( "found an integer, = %d\n",
                                  atoi( yytext ) );
                          }

              "."         printf( "found a dot\n" );

          Here is a scanner which recognizes (and discards) C comments
          while maintaining a count of the current input line.

              %x comment
              %%
                      int line_num = 1;

              "/*"         BEGIN(comment);

              <comment>[^*\n]*        /* eat anything that's not a '*' */
              <comment>"*"+[^*/\n]*   /* eat up '*'s not followed by '/'s */
              <comment>\n             ++line_num;
              <comment>"*"+"/"        BEGIN(INITIAL);

          This scanner goes to a bit of trouble to match as much text
          as possible with each rule.  In general, when attempting to
          write a high-speed scanner try to match as much possible in
          each rule, as it's a big win.

          Note that start-conditions names are really integer values
          and can be stored as such.  Thus, the above could be
          extended in the following fashion:

              %x comment foo
              %%
                      int line_num = 1;
                      int comment_caller;

              "/*"         {
                           comment_caller = INITIAL;
                           BEGIN(comment);
                           }




     Page 20                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



              ...

              <foo>"/*"    {
                           comment_caller = foo;
                           BEGIN(comment);
                           }

              <comment>[^*\n]*        /* eat anything that's not a '*' */
              <comment>"*"+[^*/\n]*   /* eat up '*'s not followed by '/'s */
              <comment>\n             ++line_num;
              <comment>"*"+"/"        BEGIN(comment_caller);

          Furthermore, you can access the current start condition
          using the integer-valued YYYYYYYY____SSSSTTTTAAAARRRRTTTT macro.  For example, the
          above assignments to _c_o_m_m_e_n_t__c_a_l_l_e_r could instead be written

              comment_caller = YY_START;

          Flex provides YYYYYYYYSSSSTTTTAAAATTTTEEEE as an alias for YYYYYYYY____SSSSTTTTAAAARRRRTTTT (since that
          is what's used by AT&T _l_e_x).

          Note that start conditions do not have their own name-space;
          %s's and %x's declare names in the same fashion as
          #define's.

          Finally, here's an example of how to match C-style quoted
          strings using exclusive start conditions, including expanded
          escape sequences (but not including checking for a string
          that's too long):

              %x str

              %%
                      char string_buf[MAX_STR_CONST];
                      char *string_buf_ptr;


              \"      string_buf_ptr = string_buf; BEGIN(str);

              <str>\"        { /* saw closing quote - all done */
                      BEGIN(INITIAL);
                      *string_buf_ptr = '\0';
                      /* return string constant token type and
                       * value to parser
                       */
                      }

              <str>\n        {
                      /* error - unterminated string constant */
                      /* generate error message */
                      }




     Page 21                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



              <str>\\[0-7]{1,3} {
                      /* octal escape sequence */
                      int result;

                      (void) sscanf( yytext + 1, "%o", &result );

                      if ( result > 0xff )
                              /* error, constant is out-of-bounds */

                      *string_buf_ptr++ = result;
                      }

              <str>\\[0-9]+ {
                      /* generate error - bad escape sequence; something
                       * like '\48' or '\0777777'
                       */
                      }

              <str>\\n  *string_buf_ptr++ = '\n';
              <str>\\t  *string_buf_ptr++ = '\t';
              <str>\\r  *string_buf_ptr++ = '\r';
              <str>\\b  *string_buf_ptr++ = '\b';
              <str>\\f  *string_buf_ptr++ = '\f';

              <str>\\(.|\n)  *string_buf_ptr++ = yytext[1];

              <str>[^\\\n\"]+        {
                      char *yptr = yytext;

                      while ( *yptr )
                              *string_buf_ptr++ = *yptr++;
                      }


          Often, such as in some of the examples above, you wind up
          writing a whole bunch of rules all preceded by the same
          start condition(s).  Flex makes this a little easier and
          cleaner by introducing a notion of start condition _s_c_o_p_e. A
          start condition scope is begun with:

              <SCs>{

          where _S_C_s is a list of one or more start conditions.  Inside
          the start condition scope, every rule automatically has the
          prefix <_S_C_s> applied to it, until a '}' which matches the
          initial '{'. So, for example,

              <ESC>{
                  "\\n"   return '\n';
                  "\\r"   return '\r';
                  "\\f"   return '\f';
                  "\\0"   return '\0';



     Page 22                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



              }

          is equivalent to:

              <ESC>"\\n"  return '\n';
              <ESC>"\\r"  return '\r';
              <ESC>"\\f"  return '\f';
              <ESC>"\\0"  return '\0';

          Start condition scopes may be nested.

          Three routines are available for manipulating stacks of
          start conditions:

          vvvvooooiiiidddd yyyyyyyy____ppppuuuusssshhhh____ssssttttaaaatttteeee((((iiiinnnntttt nnnneeeewwww____ssssttttaaaatttteeee))))
               pushes the current start condition onto the top of the
               start condition stack and switches to _n_e_w__s_t_a_t_e as
               though you had used BBBBEEEEGGGGIIIINNNN nnnneeeewwww____ssssttttaaaatttteeee (recall that start
               condition names are also integers).

          vvvvooooiiiidddd yyyyyyyy____ppppoooopppp____ssssttttaaaatttteeee(((())))
               pops the top of the stack and switches to it via BBBBEEEEGGGGIIIINNNN....

          iiiinnnntttt yyyyyyyy____ttttoooopppp____ssssttttaaaatttteeee(((())))
               returns the top of the stack without altering the
               stack's contents.

          The start condition stack grows dynamically and so has no
          built-in size limitation.  If memory is exhausted, program
          execution aborts.

          To use start condition stacks, your scanner must include a
          %%%%ooooppppttttiiiioooonnnn ssssttttaaaacccckkkk directive (see Options below).

     MMMMUUUULLLLTTTTIIIIPPPPLLLLEEEE IIIINNNNPPPPUUUUTTTT BBBBUUUUFFFFFFFFEEEERRRRSSSS
          Some scanners (such as those which support "include" files)
          require reading from several input streams.  As _f_l_e_x
          scanners do a large amount of buffering, one cannot control
          where the next input will be read from by simply writing a
          YYYYYYYY____IIIINNNNPPPPUUUUTTTT which is sensitive to the scanning context.
          YYYYYYYY____IIIINNNNPPPPUUUUTTTT is only called when the scanner reaches the end of
          its buffer, which may be a long time after scanning a
          statement such as an "include" which requires switching the
          input source.

          To negotiate these sorts of problems, _f_l_e_x provides a
          mechanism for creating and switching between multiple input
          buffers.  An input buffer is created by using:

              YY_BUFFER_STATE yy_create_buffer( FILE *file, int size )

          which takes a _F_I_L_E pointer and a size and creates a buffer



     Page 23                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          associated with the given file and large enough to hold _s_i_z_e
          characters (when in doubt, use YYYYYYYY____BBBBUUUUFFFF____SSSSIIIIZZZZEEEE for the size).
          It returns a YYYYYYYY____BBBBUUUUFFFFFFFFEEEERRRR____SSSSTTTTAAAATTTTEEEE handle, which may then be
          passed to other routines (see below).  The YYYYYYYY____BBBBUUUUFFFFFFFFEEEERRRR____SSSSTTTTAAAATTTTEEEE
          type is a pointer to an opaque ssssttttrrrruuuucccctttt yyyyyyyy____bbbbuuuuffffffffeeeerrrr____ssssttttaaaatttteeee
          structure, so you may safely initialize YY_BUFFER_STATE
          variables to ((((((((YYYYYYYY____BBBBUUUUFFFFFFFFEEEERRRR____SSSSTTTTAAAATTTTEEEE)))) 0000)))) if you wish, and also
          refer to the opaque structure in order to correctly declare
          input buffers in source files other than that of your
          scanner.  Note that the _F_I_L_E pointer in the call to
          yyyyyyyy____ccccrrrreeeeaaaatttteeee____bbbbuuuuffffffffeeeerrrr is only used as the value of _y_y_i_n seen by
          YYYYYYYY____IIIINNNNPPPPUUUUTTTT;;;; if you redefine YYYYYYYY____IIIINNNNPPPPUUUUTTTT so it no longer uses
          _y_y_i_n, then you can safely pass a nil _F_I_L_E pointer to
          yyyyyyyy____ccccrrrreeeeaaaatttteeee____bbbbuuuuffffffffeeeerrrr.... You select a particular buffer to scan
          from using:

              void yy_switch_to_buffer( YY_BUFFER_STATE new_buffer )

          switches the scanner's input buffer so subsequent tokens
          will come from _n_e_w__b_u_f_f_e_r. Note that yyyyyyyy____sssswwwwiiiittttcccchhhh____ttttoooo____bbbbuuuuffffffffeeeerrrr(((())))
          may be used by yywrap() to set things up for continued
          scanning, instead of opening a new file and pointing _y_y_i_n at
          it.  Note also that switching input sources via either
          yyyyyyyy____sssswwwwiiiittttcccchhhh____ttttoooo____bbbbuuuuffffffffeeeerrrr(((()))) or yyyyyyyywwwwrrrraaaapppp(((()))) does _n_o_t change the start
          condition.

              void yy_delete_buffer( YY_BUFFER_STATE buffer )

          is used to reclaim the storage associated with a buffer.  (
          bbbbuuuuffffffffeeeerrrr can be nil, in which case the routine does nothing.)
          You can also clear the current contents of a buffer using:

              void yy_flush_buffer( YY_BUFFER_STATE buffer )

          This function discards the buffer's contents, so the next
          time the scanner attempts to match a token from the buffer,
          it will first fill the buffer anew using YYYYYYYY____IIIINNNNPPPPUUUUTTTT....

          yyyyyyyy____nnnneeeewwww____bbbbuuuuffffffffeeeerrrr(((()))) is an alias for yyyyyyyy____ccccrrrreeeeaaaatttteeee____bbbbuuuuffffffffeeeerrrr(((()))),,,, provided
          for compatibility with the C++ use of _n_e_w and _d_e_l_e_t_e for
          creating and destroying dynamic objects.

          Finally, the YYYYYYYY____CCCCUUUURRRRRRRREEEENNNNTTTT____BBBBUUUUFFFFFFFFEEEERRRR macro returns a
          YYYYYYYY____BBBBUUUUFFFFFFFFEEEERRRR____SSSSTTTTAAAATTTTEEEE handle to the current buffer.

          Here is an example of using these features for writing a
          scanner which expands include files (the <<<<<<<<EEEEOOOOFFFF>>>>>>>> feature is
          discussed below):

              /* the "incl" state is used for picking up the name
               * of an include file
               */



     Page 24                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



              %x incl

              %{
              #define MAX_INCLUDE_DEPTH 10
              YY_BUFFER_STATE include_stack[MAX_INCLUDE_DEPTH];
              int include_stack_ptr = 0;
              %}

              %%
              include             BEGIN(incl);

              [a-z]+              ECHO;
              [^a-z\n]*\n?        ECHO;

              <incl>[ \t]*      /* eat the whitespace */
              <incl>[^ \t\n]+   { /* got the include file name */
                      if ( include_stack_ptr >= MAX_INCLUDE_DEPTH )
                          {
                          fprintf( stderr, "Includes nested too deeply" );
                          exit( 1 );
                          }

                      include_stack[include_stack_ptr++] =
                          YY_CURRENT_BUFFER;

                      yyin = fopen( yytext, "r" );

                      if ( ! yyin )
                          error( ... );

                      yy_switch_to_buffer(
                          yy_create_buffer( yyin, YY_BUF_SIZE ) );

                      BEGIN(INITIAL);
                      }

              <<EOF>> {
                      if ( --include_stack_ptr < 0 )
                          {
                          yyterminate();
                          }

                      else
                          {
                          yy_delete_buffer( YY_CURRENT_BUFFER );
                          yy_switch_to_buffer(
                               include_stack[include_stack_ptr] );
                          }
                      }

          Three routines are available for setting up input buffers
          for scanning in-memory strings instead of files.  All of



     Page 25                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          them create a new input buffer for scanning the string, and
          return a corresponding YYYYYYYY____BBBBUUUUFFFFFFFFEEEERRRR____SSSSTTTTAAAATTTTEEEE handle (which you
          should delete with yyyyyyyy____ddddeeeelllleeeetttteeee____bbbbuuuuffffffffeeeerrrr(((()))) when done with it).
          They also switch to the new buffer using
          yyyyyyyy____sssswwwwiiiittttcccchhhh____ttttoooo____bbbbuuuuffffffffeeeerrrr(((()))),,,, so the next call to yyyyyyyylllleeeexxxx(((()))) will
          start scanning the string.

          yyyyyyyy____ssssccccaaaannnn____ssssttttrrrriiiinnnngggg((((ccccoooonnnnsssstttt cccchhhhaaaarrrr ****ssssttttrrrr))))
               scans a NUL-terminated string.

          yyyyyyyy____ssssccccaaaannnn____bbbbyyyytttteeeessss((((ccccoooonnnnsssstttt cccchhhhaaaarrrr ****bbbbyyyytttteeeessss,,,, iiiinnnntttt lllleeeennnn))))
               scans _l_e_n bytes (including possibly NUL's) starting at
               location _b_y_t_e_s.

          Note that both of these functions create and scan a _c_o_p_y of
          the string or bytes.  (This may be desirable, since yyyyyyyylllleeeexxxx(((())))
          modifies the contents of the buffer it is scanning.)  You
          can avoid the copy by using:

          yyyyyyyy____ssssccccaaaannnn____bbbbuuuuffffffffeeeerrrr((((cccchhhhaaaarrrr ****bbbbaaaasssseeee,,,, yyyyyyyy____ssssiiiizzzzeeee____tttt ssssiiiizzzzeeee))))
               which scans in place the buffer starting at _b_a_s_e,
               consisting of _s_i_z_e bytes, the last two bytes of which
               _m_u_s_t be YYYYYYYY____EEEENNNNDDDD____OOOOFFFF____BBBBUUUUFFFFFFFFEEEERRRR____CCCCHHHHAAAARRRR (ASCII NUL).  These last
               two bytes are not scanned; thus, scanning consists of
               bbbbaaaasssseeee[[[[0000]]]] through bbbbaaaasssseeee[[[[ssssiiiizzzzeeee----2222]]]],,,, inclusive.

               If you fail to set up _b_a_s_e in this manner (i.e., forget
               the final two YYYYYYYY____EEEENNNNDDDD____OOOOFFFF____BBBBUUUUFFFFFFFFEEEERRRR____CCCCHHHHAAAARRRR bytes), then
               yyyyyyyy____ssssccccaaaannnn____bbbbuuuuffffffffeeeerrrr(((()))) returns a nil pointer instead of
               creating a new input buffer.

               The type yyyyyyyy____ssssiiiizzzzeeee____tttt is an integral type to which you can
               cast an integer expression reflecting the size of the
               buffer.

     EEEENNNNDDDD----OOOOFFFF----FFFFIIIILLLLEEEE RRRRUUUULLLLEEEESSSS
          The special rule "<<EOF>>" indicates actions which are to be
          taken when an end-of-file is encountered and yywrap()
          returns non-zero (i.e., indicates no further files to
          process).  The action must finish by doing one of four
          things:

          -    assigning _y_y_i_n to a new input file (in previous
               versions of flex, after doing the assignment you had to
               call the special action YYYYYYYY____NNNNEEEEWWWW____FFFFIIIILLLLEEEE;;;; this is no longer
               necessary);

          -    executing a _r_e_t_u_r_n statement;

          -    executing the special yyyyyyyytttteeeerrrrmmmmiiiinnnnaaaatttteeee(((()))) action;

          -    or, switching to a new buffer using



     Page 26                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



               yyyyyyyy____sssswwwwiiiittttcccchhhh____ttttoooo____bbbbuuuuffffffffeeeerrrr(((()))) as shown in the example above.

          <<EOF>> rules may not be used with other patterns; they may
          only be qualified with a list of start conditions.  If an
          unqualified <<EOF>> rule is given, it applies to _a_l_l start
          conditions which do not already have <<EOF>> actions.  To
          specify an <<EOF>> rule for only the initial start
          condition, use

              <INITIAL><<EOF>>


          These rules are useful for catching things like unclosed
          comments.  An example:

              %x quote
              %%

              ...other rules for dealing with quotes...

              <quote><<EOF>>   {
                       error( "unterminated quote" );
                       yyterminate();
                       }
              <<EOF>>  {
                       if ( *++filelist )
                           yyin = fopen( *filelist, "r" );
                       else
                          yyterminate();
                       }


     MMMMIIIISSSSCCCCEEEELLLLLLLLAAAANNNNEEEEOOOOUUUUSSSS MMMMAAAACCCCRRRROOOOSSSS
          The macro YYYYYYYY____UUUUSSSSEEEERRRR____AAAACCCCTTTTIIIIOOOONNNN can be defined to provide an action
          which is always executed prior to the matched rule's action.
          For example, it could be #define'd to call a routine to
          convert yytext to lower-case.  When YYYYYYYY____UUUUSSSSEEEERRRR____AAAACCCCTTTTIIIIOOOONNNN is
          invoked, the variable _y_y__a_c_t gives the number of the matched
          rule (rules are numbered starting with 1).  Suppose you want
          to profile how often each of your rules is matched.  The
          following would do the trick:

              #define YY_USER_ACTION ++ctr[yy_act]

          where _c_t_r is an array to hold the counts for the different
          rules.  Note that the macro YYYYYYYY____NNNNUUUUMMMM____RRRRUUUULLLLEEEESSSS gives the total
          number of rules (including the default rule, even if you use
          ----ssss)))),,,, so a correct declaration for _c_t_r is:

              int ctr[YY_NUM_RULES];





     Page 27                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          The macro YYYYYYYY____UUUUSSSSEEEERRRR____IIIINNNNIIIITTTT may be defined to provide an action
          which is always executed before the first scan (and before
          the scanner's internal initializations are done).  For
          example, it could be used to call a routine to read in a
          data table or open a logging file.

          The macro yyyyyyyy____sssseeeetttt____iiiinnnntttteeeerrrraaaaccccttttiiiivvvveeee((((iiiissss____iiiinnnntttteeeerrrraaaaccccttttiiiivvvveeee)))) can be used to
          control whether the current buffer is considered
          _i_n_t_e_r_a_c_t_i_v_e. An interactive buffer is processed more slowly,
          but must be used when the scanner's input source is indeed
          interactive to avoid problems due to waiting to fill buffers
          (see the discussion of the ----IIII flag below).  A non-zero value
          in the macro invocation marks the buffer as interactive, a
          zero value as non-interactive.  Note that use of this macro
          overrides %%%%ooooppppttttiiiioooonnnn aaaallllwwwwaaaayyyyssss----iiiinnnntttteeeerrrraaaaccccttttiiiivvvveeee or %%%%ooooppppttttiiiioooonnnn nnnneeeevvvveeeerrrr----
          iiiinnnntttteeeerrrraaaaccccttttiiiivvvveeee (see Options below).  yyyyyyyy____sssseeeetttt____iiiinnnntttteeeerrrraaaaccccttttiiiivvvveeee(((()))) must
          be invoked prior to beginning to scan the buffer that is (or
          is not) to be considered interactive.

          The macro yyyyyyyy____sssseeeetttt____bbbboooollll((((aaaatttt____bbbboooollll)))) can be used to control whether
          the current buffer's scanning context for the next token
          match is done as though at the beginning of a line.  A non-
          zero macro argument makes rules anchored with

          The macro YYYYYYYY____AAAATTTT____BBBBOOOOLLLL(((()))) returns true if the next token scanned
          from the current buffer will have '^' rules active, false
          otherwise.

          In the generated scanner, the actions are all gathered in
          one large switch statement and separated using YYYYYYYY____BBBBRRRREEEEAAAAKKKK,,,,
          which may be redefined.  By default, it is simply a "break",
          to separate each rule's action from the following rule's.
          Redefining YYYYYYYY____BBBBRRRREEEEAAAAKKKK allows, for example, C++ users to
          #define YY_BREAK to do nothing (while being very careful
          that every rule ends with a "break" or a "return"!) to avoid
          suffering from unreachable statement warnings where because
          a rule's action ends with "return", the YYYYYYYY____BBBBRRRREEEEAAAAKKKK is
          inaccessible.

     VVVVAAAALLLLUUUUEEEESSSS AAAAVVVVAAAAIIIILLLLAAAABBBBLLLLEEEE TTTTOOOO TTTTHHHHEEEE UUUUSSSSEEEERRRR
          This section summarizes the various values available to the
          user in the rule actions.

          -    cccchhhhaaaarrrr ****yyyyyyyytttteeeexxxxtttt holds the text of the current token.  It
               may be modified but not lengthened (you cannot append
               characters to the end).

               If the special directive %%%%aaaarrrrrrrraaaayyyy appears in the first
               section of the scanner description, then yyyyyyyytttteeeexxxxtttt is
               instead declared cccchhhhaaaarrrr yyyyyyyytttteeeexxxxtttt[[[[YYYYYYYYLLLLMMMMAAAAXXXX]]]],,,, where YYYYYYYYLLLLMMMMAAAAXXXX is a
               macro definition that you can redefine in the first
               section if you don't like the default value (generally



     Page 28                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



               8KB).  Using %%%%aaaarrrrrrrraaaayyyy results in somewhat slower
               scanners, but the value of yyyyyyyytttteeeexxxxtttt becomes immune to
               calls to _i_n_p_u_t() and _u_n_p_u_t(), which potentially destroy
               its value when yyyyyyyytttteeeexxxxtttt is a character pointer.  The
               opposite of %%%%aaaarrrrrrrraaaayyyy is %%%%ppppooooiiiinnnntttteeeerrrr,,,, which is the default.

               You cannot use %%%%aaaarrrrrrrraaaayyyy when generating C++ scanner
               classes (the ----++++ flag).

          -    iiiinnnntttt yyyyyyyylllleeeennnngggg holds the length of the current token.

          -    FFFFIIIILLLLEEEE ****yyyyyyyyiiiinnnn is the file which by default _f_l_e_x reads
               from.  It may be redefined but doing so only makes
               sense before scanning begins or after an EOF has been
               encountered.  Changing it in the midst of scanning will
               have unexpected results since _f_l_e_x buffers its input;
               use yyyyyyyyrrrreeeessssttttaaaarrrrtttt(((()))) instead.  Once scanning terminates
               because an end-of-file has been seen, you can assign
               _y_y_i_n at the new input file and then call the scanner
               again to continue scanning.

          -    vvvvooooiiiidddd yyyyyyyyrrrreeeessssttttaaaarrrrtttt(((( FFFFIIIILLLLEEEE ****nnnneeeewwww____ffffiiiilllleeee )))) may be called to point
               _y_y_i_n at the new input file.  The switch-over to the new
               file is immediate (any previously buffered-up input is
               lost).  Note that calling yyyyyyyyrrrreeeessssttttaaaarrrrtttt(((()))) with _y_y_i_n as an
               argument thus throws away the current input buffer and
               continues scanning the same input file.

          -    FFFFIIIILLLLEEEE ****yyyyyyyyoooouuuutttt is the file to which EEEECCCCHHHHOOOO actions are done.
               It can be reassigned by the user.

          -    YYYYYYYY____CCCCUUUURRRRRRRREEEENNNNTTTT____BBBBUUUUFFFFFFFFEEEERRRR returns a YYYYYYYY____BBBBUUUUFFFFFFFFEEEERRRR____SSSSTTTTAAAATTTTEEEE handle to
               the current buffer.

          -    YYYYYYYY____SSSSTTTTAAAARRRRTTTT returns an integer value corresponding to the
               current start condition.  You can subsequently use this
               value with BBBBEEEEGGGGIIIINNNN to return to that start condition.

     IIIINNNNTTTTEEEERRRRFFFFAAAACCCCIIIINNNNGGGG WWWWIIIITTTTHHHH YYYYAAAACCCCCCCC
          One of the main uses of _f_l_e_x is as a companion to the _y_a_c_c
          parser-generator.  _y_a_c_c parsers expect to call a routine
          named yyyyyyyylllleeeexxxx(((()))) to find the next input token.  The routine is
          supposed to return the type of the next token as well as
          putting any associated value in the global yyyyyyyyllllvvvvaaaallll.... To use
          _f_l_e_x with _y_a_c_c, one specifies the ----dddd option to _y_a_c_c to
          instruct it to generate the file yyyy....ttttaaaabbbb....hhhh containing
          definitions of all the %%%%ttttooookkkkeeeennnnssss appearing in the _y_a_c_c input.
          This file is then included in the _f_l_e_x scanner.  For
          example, if one of the tokens is "TOK_NUMBER", part of the
          scanner might look like:

              %{



     Page 29                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



              #include "y.tab.h"
              %}

              %%

              [0-9]+        yylval = atoi( yytext ); return TOK_NUMBER;


     OOOOPPPPTTTTIIIIOOOONNNNSSSS
          _f_l_e_x has the following options:

          ----bbbb   Generate backing-up information to _l_e_x._b_a_c_k_u_p. This is
               a list of scanner states which require backing up and
               the input characters on which they do so.  By adding
               rules one can remove backing-up states.  If _a_l_l
               backing-up states are eliminated and ----CCCCffff or ----CCCCFFFF is
               used, the generated scanner will run faster (see the ----pppp
               flag).  Only users who wish to squeeze every last cycle
               out of their scanners need worry about this option.
               (See the section on Performance Considerations below.)

          ----cccc   is a do-nothing, deprecated option included for POSIX
               compliance.

          ----dddd   makes the generated scanner run in _d_e_b_u_g mode.
               Whenever a pattern is recognized and the global
               yyyyyyyy____fffflllleeeexxxx____ddddeeeebbbbuuuugggg is non-zero (which is the default), the
               scanner will write to _s_t_d_e_r_r a line of the form:

                   --accepting rule at line 53 ("the matched text")

               The line number refers to the location of the rule in
               the file defining the scanner (i.e., the file that was
               fed to flex).  Messages are also generated when the
               scanner backs up, accepts the default rule, reaches the
               end of its input buffer (or encounters a NUL; at this
               point, the two look the same as far as the scanner's
               concerned), or reaches an end-of-file.

          ----ffff   specifies _f_a_s_t _s_c_a_n_n_e_r. No table compression is done
               and stdio is bypassed.  The result is large but fast.
               This option is equivalent to ----CCCCffffrrrr (see below).

          ----hhhh   generates a "help" summary of _f_l_e_x'_s options to _s_t_d_o_u_t
               and then exits.  ----???? and --------hhhheeeellllpppp are synonyms for ----hhhh....

          ----iiii   instructs _f_l_e_x to generate a _c_a_s_e-_i_n_s_e_n_s_i_t_i_v_e scanner.
               The case of letters given in the _f_l_e_x input patterns
               will be ignored, and tokens in the input will be
               matched regardless of case.  The matched text given in
               _y_y_t_e_x_t will have the preserved case (i.e., it will not
               be folded).



     Page 30                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          ----llll   turns on maximum compatibility with the original AT&T
               _l_e_x implementation.  Note that this does not mean _f_u_l_l
               compatibility.  Use of this option costs a considerable
               amount of performance, and it cannot be used with the
               ----++++,,,, ----ffff,,,, ----FFFF,,,, ----CCCCffff,,,, or ----CCCCFFFF options.  For details on the
               compatibilities it provides, see the section
               "Incompatibilities With Lex And POSIX" below.  This
               option also results in the name YYYYYYYY____FFFFLLLLEEEEXXXX____LLLLEEEEXXXX____CCCCOOOOMMMMPPPPAAAATTTT
               being #define'd in the generated scanner.

          ----nnnn   is another do-nothing, deprecated option included only
               for POSIX compliance.

          ----pppp   generates a performance report to stderr.  The report
               consists of comments regarding features of the _f_l_e_x
               input file which will cause a serious loss of
               performance in the resulting scanner.  If you give the
               flag twice, you will also get comments regarding
               features that lead to minor performance losses.

               Note that the use of RRRREEEEJJJJEEEECCCCTTTT,,,, %%%%ooooppppttttiiiioooonnnn yyyyyyyylllliiiinnnneeeennnnoooo,,,, and
               variable trailing context (see the Deficiencies / Bugs
               section below) entails a substantial performance
               penalty; use of _y_y_m_o_r_e(), the ^^^^ operator, and the ----IIII
               flag entail minor performance penalties.

          ----ssss   causes the _d_e_f_a_u_l_t _r_u_l_e (that unmatched scanner input
               is echoed to _s_t_d_o_u_t) to be suppressed.  If the scanner
               encounters input that does not match any of its rules,
               it aborts with an error.  This option is useful for
               finding holes in a scanner's rule set.

          ----tttt   instructs _f_l_e_x to write the scanner it generates to
               standard output instead of lllleeeexxxx....yyyyyyyy....cccc....

          ----vvvv   specifies that _f_l_e_x should write to _s_t_d_e_r_r a summary of
               statistics regarding the scanner it generates.  Most of
               the statistics are meaningless to the casual _f_l_e_x user,
               but the first line identifies the version of _f_l_e_x (same
               as reported by ----VVVV)))),,,, and the next line the flags used
               when generating the scanner, including those that are
               on by default.

          ----wwww   suppresses warning messages.

          ----BBBB   instructs _f_l_e_x to generate a _b_a_t_c_h scanner, the
               opposite of _i_n_t_e_r_a_c_t_i_v_e scanners generated by ----IIII (see
               below).  In general, you use ----BBBB when you are _c_e_r_t_a_i_n
               that your scanner will never be used interactively, and
               you want to squeeze a _l_i_t_t_l_e more performance out of
               it.  If your goal is instead to squeeze out a _l_o_t more
               performance, you should  be using the ----CCCCffff or ----CCCCFFFF



     Page 31                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



               options (discussed below), which turn on ----BBBB
               automatically anyway.

          ----FFFF   specifies that the _f_a_s_t scanner table representation
               should be used (and stdio bypassed).  This
               representation is about as fast as the full table
               representation ((((----ffff)))),,,, and for some sets of patterns will
               be considerably smaller (and for others, larger).  In
               general, if the pattern set contains both "keywords"
               and a catch-all, "identifier" rule, such as in the set:

                   "case"    return TOK_CASE;
                   "switch"  return TOK_SWITCH;
                   ...
                   "default" return TOK_DEFAULT;
                   [a-z]+    return TOK_ID;

               then you're better off using the full table
               representation.  If only the "identifier" rule is
               present and you then use a hash table or some such to
               detect the keywords, you're better off using ----FFFF....

               This option is equivalent to ----CCCCFFFFrrrr (see below).  It
               cannot be used with ----++++....

          ----IIII   instructs _f_l_e_x to generate an _i_n_t_e_r_a_c_t_i_v_e scanner.  An
               interactive scanner is one that only looks ahead to
               decide what token has been matched if it absolutely
               must.  It turns out that always looking one extra
               character ahead, even if the scanner has already seen
               enough text to disambiguate the current token, is a bit
               faster than only looking ahead when necessary.  But
               scanners that always look ahead give dreadful
               interactive performance; for example, when a user types
               a newline, it is not recognized as a newline token
               until they enter _a_n_o_t_h_e_r token, which often means
               typing in another whole line.

               _F_l_e_x scanners default to _i_n_t_e_r_a_c_t_i_v_e unless you use the
               ----CCCCffff or ----CCCCFFFF table-compression options (see below).
               That's because if you're looking for high-performance
               you should be using one of these options, so if you
               didn't, _f_l_e_x assumes you'd rather trade off a bit of
               run-time performance for intuitive interactive
               behavior.  Note also that you _c_a_n_n_o_t use ----IIII in
               conjunction with ----CCCCffff or ----CCCCFFFF.... Thus, this option is not
               really needed; it is on by default for all those cases
               in which it is allowed.

               You can force a scanner to _n_o_t be interactive by using
               ----BBBB (see above).




     Page 32                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          ----LLLL   instructs _f_l_e_x not to generate ####lllliiiinnnneeee directives.
               Without this option, _f_l_e_x peppers the generated scanner
               with #line directives so error messages in the actions
               will be correctly located with respect to either the
               original _f_l_e_x input file (if the errors are due to code
               in the input file), or lllleeeexxxx....yyyyyyyy....cccc (if the errors are
               _f_l_e_x'_s fault -- you should report these sorts of errors
               to the email address given below).

          ----TTTT   makes _f_l_e_x run in _t_r_a_c_e mode.  It will generate a lot
               of messages to _s_t_d_e_r_r concerning the form of the input
               and the resultant non-deterministic and deterministic
               finite automata.  This option is mostly for use in
               maintaining _f_l_e_x.

          ----VVVV   prints the version number to _s_t_d_o_u_t and exits.
               --------vvvveeeerrrrssssiiiioooonnnn is a synonym for ----VVVV....

          --------ppppoooossssiiiixxxx----ccccoooommmmppppaaaatttt
               turns on maximum compatibility with the POSIX 1003.2-
               1992 definition of _l_e_x (----XXXX is a synonym for
               --------ppppoooossssiiiixxxx----ccccoooommmmppppaaaatttt).  Since _f_l_e_x was originally designed to
               implement the POSIX definition of _l_e_x this generally
               involves very few changes in behavior.  At the current
               writing the known differences between _f_l_e_x and the
               POSIX standard are:

               +    In POSIX and AT&T _l_e_x, the repeat operator, {{{{}}}},
                    has lower precedence than concatenation (thus
                    aaaabbbb{{{{3333}}}} yields aaaabbbbaaaabbbbaaaabbbb).  Most POSIX utilities use an
                    Extended Regular Expression (ERE) precedence that
                    has the precedence of the repeat operator higher
                    than concatenation (which causes aaaabbbb{{{{3333}}}} to yield
                    aaaabbbbbbbbbbbb).  By default, _f_l_e_x places the precedence of
                    the repeat operator higher than concatenation
                    which matches the ERE processing of other POSIX
                    utilities.  When either --------ppppoooossssiiiixxxx----ccccoooommmmppppaaaatttt or ----llll are
                    specified, _f_l_e_x will use the traditional AT&T and
                    POSIX-compliant precedence for the repeat operator
                    where concatenation has higher precedence than the
                    repeat operator.

          ----7777   instructs _f_l_e_x to generate a 7-bit scanner, i.e., one
               which can only recognized 7-bit characters in its
               input.  The advantage of using ----7777 is that the scanner's
               tables can be up to half the size of those generated
               using the ----8888 option (see below).  The disadvantage is
               that such scanners often hang or crash if their input
               contains an 8-bit character.

               Note, however, that unless you generate your scanner
               using the ----CCCCffff or ----CCCCFFFF table compression options, use of



     Page 33                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



               ----7777 will save only a small amount of table space, and
               make your scanner considerably less portable.  _F_l_e_x'_s
               default behavior is to generate an 8-bit scanner unless
               you use the ----CCCCffff or ----CCCCFFFF,,,, in which case _f_l_e_x defaults to
               generating 7-bit scanners unless your site was always
               configured to generate 8-bit scanners (as will often be
               the case with non-USA sites).  You can tell whether
               flex generated a 7-bit or an 8-bit scanner by
               inspecting the flag summary in the ----vvvv output as
               described above.

               Note that if you use ----CCCCffffeeee or ----CCCCFFFFeeee (those table
               compression options, but also using equivalence classes
               as discussed see below), flex still defaults to
               generating an 8-bit scanner, since usually with these
               compression options full 8-bit tables are not much more
               expensive than 7-bit tables.

          ----8888   instructs _f_l_e_x to generate an 8-bit scanner, i.e., one
               which can recognize 8-bit characters.  This flag is
               only needed for scanners generated using ----CCCCffff or ----CCCCFFFF,,,, as
               otherwise flex defaults to generating an 8-bit scanner
               anyway.

               See the discussion of ----7777 above for flex's default
               behavior and the tradeoffs between 7-bit and 8-bit
               scanners.

          ----++++   specifies that you want flex to generate a C++ scanner
               class.  See the section on Generating C++ Scanners
               below for details.

          ----CCCC[[[[aaaaeeeeffffFFFFmmmmrrrr]]]]
               controls the degree of table compression and, more
               generally, trade-offs between small scanners and fast
               scanners.

               ----CCCCaaaa ("align") instructs flex to trade off larger tables
               in the generated scanner for faster performance because
               the elements of the tables are better aligned for
               memory access and computation.  On some RISC
               architectures, fetching and manipulating longwords is
               more efficient than with smaller-sized units such as
               shortwords.  This option can double the size of the
               tables used by your scanner.

               ----CCCCeeee directs _f_l_e_x to construct _e_q_u_i_v_a_l_e_n_c_e _c_l_a_s_s_e_s,
               i.e., sets of characters which have identical lexical
               properties (for example, if the only appearance of
               digits in the _f_l_e_x input is in the character class
               "[0-9]" then the digits '0', '1', ..., '9' will all be
               put in the same equivalence class).  Equivalence



     Page 34                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



               classes usually give dramatic reductions in the final
               table/object file sizes (typically a factor of 2-5) and
               are pretty cheap performance-wise (one array look-up
               per character scanned).

               ----CCCCffff specifies that the _f_u_l_l scanner tables should be
               generated - _f_l_e_x should not compress the tables by
               taking advantages of similar transition functions for
               different states.

               ----CCCCFFFF specifies that the alternate fast scanner
               representation (described above under the ----FFFF flag)
               should be used.  This option cannot be used with ----++++....

               ----CCCCmmmm directs _f_l_e_x to construct _m_e_t_a-_e_q_u_i_v_a_l_e_n_c_e _c_l_a_s_s_e_s,
               which are sets of equivalence classes (or characters,
               if equivalence classes are not being used) that are
               commonly used together.  Meta-equivalence classes are
               often a big win when using compressed tables, but they
               have a moderate performance impact (one or two "if"
               tests and one array look-up per character scanned).

               ----CCCCrrrr causes the generated scanner to _b_y_p_a_s_s use of the
               standard I/O library (stdio) for input.  Instead of
               calling ffffrrrreeeeaaaadddd(((()))) or ggggeeeettttcccc(((()))),,,, the scanner will use the
               rrrreeeeaaaadddd(((()))) system call, resulting in a performance gain
               which varies from system to system, but in general is
               probably negligible unless you are also using ----CCCCffff or
               ----CCCCFFFF.... Using ----CCCCrrrr can cause strange behavior if, for
               example, you read from _y_y_i_n using stdio prior to
               calling the scanner (because the scanner will miss
               whatever text your previous reads left in the stdio
               input buffer).

               ----CCCCrrrr has no effect if you define YYYYYYYY____IIIINNNNPPPPUUUUTTTT (see The
               Generated Scanner above).

               A lone ----CCCC specifies that the scanner tables should be
               compressed but neither equivalence classes nor meta-
               equivalence classes should be used.

               The options ----CCCCffff or ----CCCCFFFF and ----CCCCmmmm do not make sense
               together - there is no opportunity for meta-equivalence
               classes if the table is not being compressed.
               Otherwise the options may be freely mixed, and are
               cumulative.

               The default setting is ----CCCCeeeemmmm,,,, which specifies that _f_l_e_x
               should generate equivalence classes and meta-
               equivalence classes.  This setting provides the highest
               degree of table compression.  You can trade off
               faster-executing scanners at the cost of larger tables



     Page 35                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



               with the following generally being true:

                   slowest & smallest
                         -Cem
                         -Cm
                         -Ce
                         -C
                         -C{f,F}e
                         -C{f,F}
                         -C{f,F}a
                   fastest & largest

               Note that scanners with the smallest tables are usually
               generated and compiled the quickest, so during
               development you will usually want to use the default,
               maximal compression.

               ----CCCCffffeeee is often a good compromise between speed and size
               for production scanners.

          ----oooooooouuuuttttppppuuuutttt
               directs flex to write the scanner to the file oooouuuuttttppppuuuutttt
               instead of lllleeeexxxx....yyyyyyyy....cccc.... If you combine ----oooo with the ----tttt
               option, then the scanner is written to _s_t_d_o_u_t but its
               ####lllliiiinnnneeee directives (see the ----LLLL option above) refer to the
               file oooouuuuttttppppuuuutttt....

          ----PPPPpppprrrreeeeffffiiiixxxx
               changes the default _y_y prefix used by _f_l_e_x for all
               globally-visible variable and function names to instead
               be _p_r_e_f_i_x. For example, ----PPPPffffoooooooo changes the name of
               yyyyyyyytttteeeexxxxtttt to ffffooooooootttteeeexxxxtttt.... It also changes the name of the
               default output file from lllleeeexxxx....yyyyyyyy....cccc to lllleeeexxxx....ffffoooooooo....cccc.... Here
               are all of the names affected:

                   yy_create_buffer
                   yy_delete_buffer
                   yy_flex_debug
                   yy_init_buffer
                   yy_flush_buffer
                   yy_load_buffer_state
                   yy_switch_to_buffer
                   yyin
                   yyleng
                   yylex
                   yylineno
                   yyout
                   yyrestart
                   yytext
                   yywrap

               (If you are using a C++ scanner, then only yyyyyyyywwwwrrrraaaapppp and



     Page 36                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



               yyyyyyyyFFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr are affected.)  Within your scanner itself,
               you can still refer to the global variables and
               functions using either version of their name; but
               externally, they have the modified name.

               This option lets you easily link together multiple _f_l_e_x
               programs into the same executable.  Note, though, that
               using this option also renames yyyyyyyywwwwrrrraaaapppp(((()))),,,, so you now
               _m_u_s_t either provide your own (appropriately-named)
               version of the routine for your scanner, or use %%%%ooooppppttttiiiioooonnnn
               nnnnooooyyyyyyyywwwwrrrraaaapppp,,,, as linking with ----llllffffllll no longer provides one
               for you by default.

          ----SSSSsssskkkkeeeelllleeeettttoooonnnn____ffffiiiilllleeee
               overrides the default skeleton file from which _f_l_e_x
               constructs its scanners.  You'll never need this option
               unless you are doing _f_l_e_x maintenance or development.

          _f_l_e_x also provides a mechanism for controlling options
          within the scanner specification itself, rather than from
          the flex command-line.  This is done by including %%%%ooooppppttttiiiioooonnnn
          directives in the first section of the scanner
          specification.  You can specify multiple options with a
          single %%%%ooooppppttttiiiioooonnnn directive, and multiple directives in the
          first section of your flex input file.

          Most options are given simply as names, optionally preceded
          by the word "no" (with no intervening whitespace) to negate
          their meaning.  A number are equivalent to flex flags or
          their negation:

              7bit            -7 option
              8bit            -8 option
              align           -Ca option
              backup          -b option
              batch           -B option
              c++             -+ option

              caseful or
              case-sensitive  opposite of -i (default)

              case-insensitive or
              caseless        -i option

              debug           -d option
              default         opposite of -s option
              ecs             -Ce option
              fast            -F option
              full            -f option
              interactive     -I option
              lex-compat      -l option
              meta-ecs        -Cm option



     Page 37                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



              perf-report     -p option
              read            -Cr option
              stdout          -t option
              verbose         -v option
              warn            opposite of -w option
                              (use "%option nowarn" for -w)

              array           equivalent to "%array"
              pointer         equivalent to "%pointer" (default)

          Some %%%%ooooppppttttiiiioooonnnn''''ssss provide features otherwise not available:

          aaaallllwwwwaaaayyyyssss----iiiinnnntttteeeerrrraaaaccccttttiiiivvvveeee
               instructs flex to generate a scanner which always
               considers its input "interactive".  Normally, on each
               new input file the scanner calls iiiissssaaaattttttttyyyy(((()))) in an attempt
               to determine whether the scanner's input source is
               interactive and thus should be read a character at a
               time.  When this option is used, however, then no such
               call is made.

          mmmmaaaaiiiinnnn directs flex to provide a default mmmmaaaaiiiinnnn(((()))) program for
               the scanner, which simply calls yyyyyyyylllleeeexxxx(((()))).... This option
               implies nnnnooooyyyyyyyywwwwrrrraaaapppp (see below).

          nnnneeeevvvveeeerrrr----iiiinnnntttteeeerrrraaaaccccttttiiiivvvveeee
               instructs flex to generate a scanner which never
               considers its input "interactive" (again, no call made
               to iiiissssaaaattttttttyyyy(((()))))))).... This is the opposite of aaaallllwwwwaaaayyyyssss----
               iiiinnnntttteeeerrrraaaaccccttttiiiivvvveeee....

          ssssttttaaaacccckkkk
               enables the use of start condition stacks (see Start
               Conditions above).

          ssssttttddddiiiinnnniiiitttt
               if set (i.e., %%%%ooooppppttttiiiioooonnnn ssssttttddddiiiinnnniiiitttt)))) initializes _y_y_i_n and
               _y_y_o_u_t to _s_t_d_i_n and _s_t_d_o_u_t, instead of the default of
               _n_i_l. Some existing _l_e_x programs depend on this
               behavior, even though it is not compliant with ANSI C,
               which does not require _s_t_d_i_n and _s_t_d_o_u_t to be compile-
               time constant.

          yyyyyyyylllliiiinnnneeeennnnoooo
               directs _f_l_e_x to generate a scanner that maintains the
               number of the current line read from its input in the
               global variable yyyyyyyylllliiiinnnneeeennnnoooo.... This option is implied by
               %%%%ooooppppttttiiiioooonnnn lllleeeexxxx----ccccoooommmmppppaaaatttt....

          yyyyyyyywwwwrrrraaaapppp
               if unset (i.e., %%%%ooooppppttttiiiioooonnnn nnnnooooyyyyyyyywwwwrrrraaaapppp)))),,,, makes the scanner
               not call yyyyyyyywwwwrrrraaaapppp(((()))) upon an end-of-file, but simply



     Page 38                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



               assume that there are no more files to scan (until the
               user points _y_y_i_n at a new file and calls yyyyyyyylllleeeexxxx(((())))
               again).

          _f_l_e_x scans your rule actions to determine whether you use
          the RRRREEEEJJJJEEEECCCCTTTT or yyyyyyyymmmmoooorrrreeee(((()))) features.  The rrrreeeejjjjeeeecccctttt and yyyyyyyymmmmoooorrrreeee
          options are available to override its decision as to whether
          you use the options, either by setting them (e.g., %%%%ooooppppttttiiiioooonnnn
          rrrreeeejjjjeeeecccctttt)))) to indicate the feature is indeed used, or unsetting
          them to indicate it actually is not used (e.g., %%%%ooooppppttttiiiioooonnnn
          nnnnooooyyyyyyyymmmmoooorrrreeee))))....

          Three options take string-delimited values, offset with '=':

              %option outfile="ABC"

          is equivalent to ----ooooAAAABBBBCCCC,,,, and

              %option prefix="XYZ"

          is equivalent to ----PPPPXXXXYYYYZZZZ.... Finally,

              %option yyclass="foo"

          only applies when generating a C++ scanner ( ----++++ option).  It
          informs _f_l_e_x that you have derived ffffoooooooo as a subclass of
          yyyyyyyyFFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr,,,, so _f_l_e_x will place your actions in the member
          function ffffoooooooo::::::::yyyyyyyylllleeeexxxx(((()))) instead of yyyyyyyyFFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr::::::::yyyyyyyylllleeeexxxx(((()))).... It
          also generates a yyyyyyyyFFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr::::::::yyyyyyyylllleeeexxxx(((()))) member function that
          emits a run-time error (by invoking
          yyyyyyyyFFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr::::::::LLLLeeeexxxxeeeerrrrEEEErrrrrrrroooorrrr(((()))))))) if called.  See Generating C++
          Scanners, below, for additional information.

          A number of options are available for lint purists who want
          to suppress the appearance of unneeded routines in the
          generated scanner.  Each of the following, if unset (e.g.,
          %%%%ooooppppttttiiiioooonnnn nnnnoooouuuunnnnppppuuuutttt ), results in the corresponding routine not
          appearing in the generated scanner:

              input, unput
              yy_push_state, yy_pop_state, yy_top_state
              yy_scan_buffer, yy_scan_bytes, yy_scan_string

          (though yyyyyyyy____ppppuuuusssshhhh____ssssttttaaaatttteeee(((()))) and friends won't appear anyway
          unless you use %%%%ooooppppttttiiiioooonnnn ssssttttaaaacccckkkk))))....

     PPPPEEEERRRRFFFFOOOORRRRMMMMAAAANNNNCCCCEEEE CCCCOOOONNNNSSSSIIIIDDDDEEEERRRRAAAATTTTIIIIOOOONNNNSSSS
          The main design goal of _f_l_e_x is that it generate high-
          performance scanners.  It has been optimized for dealing
          well with large sets of rules.  Aside from the effects on
          scanner speed of the table compression ----CCCC options outlined
          above, there are a number of options/actions which degrade



     Page 39                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          performance.  These are, from most expensive to least:

              REJECT
              %option yylineno
              arbitrary trailing context

              pattern sets that require backing up
              %array
              %option interactive
              %option always-interactive

              '^' beginning-of-line operator
              yymore()

          with the first three all being quite expensive and the last
          two being quite cheap.  Note also that uuuunnnnppppuuuutttt(((()))) is
          implemented as a routine call that potentially does quite a
          bit of work, while yyyyyyyylllleeeessssssss(((()))) is a quite-cheap macro; so if
          just putting back some excess text you scanned, use
          yyyyyyyylllleeeessssssss(((())))....

          RRRREEEEJJJJEEEECCCCTTTT should be avoided at all costs when performance is
          important.  It is a particularly expensive option.

          Getting rid of backing up is messy and often may be an
          enormous amount of work for a complicated scanner.  In
          principal, one begins by using the ----bbbb flag to generate a
          _l_e_x._b_a_c_k_u_p file.  For example, on the input

              %%
              foo        return TOK_KEYWORD;
              foobar     return TOK_KEYWORD;

          the file looks like:

              State #6 is non-accepting -
               associated rule line numbers:
                     2       3
               out-transitions: [ o ]
               jam-transitions: EOF [ \001-n  p-\177 ]

              State #8 is non-accepting -
               associated rule line numbers:
                     3
               out-transitions: [ a ]
               jam-transitions: EOF [ \001-`  b-\177 ]

              State #9 is non-accepting -
               associated rule line numbers:
                     3
               out-transitions: [ r ]
               jam-transitions: EOF [ \001-q  s-\177 ]



     Page 40                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



              Compressed tables always back up.

          The first few lines tell us that there's a scanner state in
          which it can make a transition on an 'o' but not on any
          other character, and that in that state the currently
          scanned text does not match any rule.  The state occurs when
          trying to match the rules found at lines 2 and 3 in the
          input file.  If the scanner is in that state and then reads
          something other than an 'o', it will have to back up to find
          a rule which is matched.  With a bit of headscratching one
          can see that this must be the state it's in when it has seen
          "fo".  When this has happened, if anything other than
          another 'o' is seen, the scanner will have to back up to
          simply match the 'f' (by the default rule).

          The comment regarding State #8 indicates there's a problem
          when "foob" has been scanned.  Indeed, on any character
          other than an 'a', the scanner will have to back up to
          accept "foo".  Similarly, the comment for State #9 concerns
          when "fooba" has been scanned and an 'r' does not follow.

          The final comment reminds us that there's no point going to
          all the trouble of removing backing up from the rules unless
          we're using ----CCCCffff or ----CCCCFFFF,,,, since there's no performance gain
          doing so with compressed scanners.

          The way to remove the backing up is to add "error" rules:

              %%
              foo         return TOK_KEYWORD;
              foobar      return TOK_KEYWORD;

              fooba       |
              foob        |
              fo          {
                          /* false alarm, not really a keyword */
                          return TOK_ID;
                          }


          Eliminating backing up among a list of keywords can also be
          done using a "catch-all" rule:

              %%
              foo         return TOK_KEYWORD;
              foobar      return TOK_KEYWORD;

              [a-z]+      return TOK_ID;

          This is usually the best solution when appropriate.

          Backing up messages tend to cascade.  With a complicated set



     Page 41                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          of rules it's not uncommon to get hundreds of messages.  If
          one can decipher them, though, it often only takes a dozen
          or so rules to eliminate the backing up (though it's easy to
          make a mistake and have an error rule accidentally match a
          valid token.  A possible future _f_l_e_x feature will be to
          automatically add rules to eliminate backing up).

          It's important to keep in mind that you gain the benefits of
          eliminating backing up only if you eliminate _e_v_e_r_y instance
          of backing up.  Leaving just one means you gain nothing.

          _V_a_r_i_a_b_l_e trailing context (where both the leading and
          trailing parts do not have a fixed length) entails almost
          the same performance loss as RRRREEEEJJJJEEEECCCCTTTT (i.e., substantial).  So
          when possible a rule like:

              %%
              mouse|rat/(cat|dog)   run();

          is better written:

              %%
              mouse/cat|dog         run();
              rat/cat|dog           run();

          or as

              %%
              mouse|rat/cat         run();
              mouse|rat/dog         run();

          Note that here the special '|' action does _n_o_t provide any
          savings, and can even make things worse (see Deficiencies /
          Bugs below).

          Another area where the user can increase a scanner's
          performance (and one that's easier to implement) arises from
          the fact that the longer the tokens matched, the faster the
          scanner will run.  This is because with long tokens the
          processing of most input characters takes place in the
          (short) inner scanning loop, and does not often have to go
          through the additional work of setting up the scanning
          environment (e.g., yyyyyyyytttteeeexxxxtttt)))) for the action.  Recall the
          scanner for C comments:

              %x comment
              %%
                      int line_num = 1;

              "/*"         BEGIN(comment);

              <comment>[^*\n]*



     Page 42                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



              <comment>"*"+[^*/\n]*
              <comment>\n             ++line_num;
              <comment>"*"+"/"        BEGIN(INITIAL);

          This could be sped up by writing it as:

              %x comment
              %%
                      int line_num = 1;

              "/*"         BEGIN(comment);

              <comment>[^*\n]*
              <comment>[^*\n]*\n      ++line_num;
              <comment>"*"+[^*/\n]*
              <comment>"*"+[^*/\n]*\n ++line_num;
              <comment>"*"+"/"        BEGIN(INITIAL);

          Now instead of each newline requiring the processing of
          another action, recognizing the newlines is "distributed"
          over the other rules to keep the matched text as long as
          possible.  Note that _a_d_d_i_n_g rules does _n_o_t slow down the
          scanner!  The speed of the scanner is independent of the
          number of rules or (modulo the considerations given at the
          beginning of this section) how complicated the rules are
          with regard to operators such as '*' and '|'.

          A final example in speeding up a scanner: suppose you want
          to scan through a file containing identifiers and keywords,
          one per line and with no other extraneous characters, and
          recognize all the keywords.  A natural first approach is:

              %%
              asm      |
              auto     |
              break    |
              ... etc ...
              volatile |
              while    /* it's a keyword */

              .|\n     /* it's not a keyword */

          To eliminate the back-tracking, introduce a catch-all rule:

              %%
              asm      |
              auto     |
              break    |
              ... etc ...
              volatile |
              while    /* it's a keyword */




     Page 43                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



              [a-z]+   |
              .|\n     /* it's not a keyword */

          Now, if it's guaranteed that there's exactly one word per
          line, then we can reduce the total number of matches by a
          half by merging in the recognition of newlines with that of
          the other tokens:

              %%
              asm\n    |
              auto\n   |
              break\n  |
              ... etc ...
              volatile\n |
              while\n  /* it's a keyword */

              [a-z]+\n |
              .|\n     /* it's not a keyword */

          One has to be careful here, as we have now reintroduced
          backing up into the scanner.  In particular, while _w_e know
          that there will never be any characters in the input stream
          other than letters or newlines, _f_l_e_x can't figure this out,
          and it will plan for possibly needing to back up when it has
          scanned a token like "auto" and then the next character is
          something other than a newline or a letter.  Previously it
          would then just match the "auto" rule and be done, but now
          it has no "auto" rule, only a "auto\n" rule.  To eliminate
          the possibility of backing up, we could either duplicate all
          rules but without final newlines, or, since we never expect
          to encounter such an input and therefore don't how it's
          classified, we can introduce one more catch-all rule, this
          one which doesn't include a newline:

              %%
              asm\n    |
              auto\n   |
              break\n  |
              ... etc ...
              volatile\n |
              while\n  /* it's a keyword */

              [a-z]+\n |
              [a-z]+   |
              .|\n     /* it's not a keyword */

          Compiled with ----CCCCffff,,,, this is about as fast as one can get a
          _f_l_e_x scanner to go for this particular problem.

          A final note:  _f_l_e_x is slow when matching NUL's,
          particularly when a token contains multiple NUL's.  It's
          best to write rules which match _s_h_o_r_t amounts of text if



     Page 44                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          it's anticipated that the text will often include NUL's.

          Another final note regarding performance: as mentioned above
          in the section How the Input is Matched, dynamically
          resizing yyyyyyyytttteeeexxxxtttt to accommodate huge tokens is a slow process
          because it presently requires that the (huge) token be
          rescanned from the beginning.  Thus if performance is vital,
          you should attempt to match "large" quantities of text but
          not "huge" quantities, where the cutoff between the two is
          at about 8K characters/token.

     GGGGEEEENNNNEEEERRRRAAAATTTTIIIINNNNGGGG CCCC++++++++ SSSSCCCCAAAANNNNNNNNEEEERRRRSSSS
          _f_l_e_x provides two different ways to generate scanners for
          use with C++.  The first way is to simply compile a scanner
          generated by _f_l_e_x using a C++ compiler instead of a C
          compiler.  You should not encounter any compilations errors
          (please report any you find to the email address given in
          the Author section below).  You can then use C++ code in
          your rule actions instead of C code.  Note that the default
          input source for your scanner remains _y_y_i_n, and default
          echoing is still done to _y_y_o_u_t. Both of these remain _F_I_L_E *
          variables and not C++ _s_t_r_e_a_m_s.

          You can also use _f_l_e_x to generate a C++ scanner class, using
          the ----++++ option (or, equivalently, %%%%ooooppppttttiiiioooonnnn cccc++++++++)))),,,, which is
          automatically specified if the name of the flex executable
          ends in a '+', such as _f_l_e_x++. When using this option, flex
          defaults to generating the scanner to the file lllleeeexxxx....yyyyyyyy....cccccccc
          instead of lllleeeexxxx....yyyyyyyy....cccc.... The generated scanner includes the
          header file _F_l_e_x_L_e_x_e_r._h, which defines the interface to two
          C++ classes.

          The first class, FFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr,,,, provides an abstract base class
          defining the general scanner class interface.  It provides
          the following member functions:

          ccccoooonnnnsssstttt cccchhhhaaaarrrr**** YYYYYYYYTTTTeeeexxxxtttt(((())))
               returns the text of the most recently matched token,
               the equivalent of yyyyyyyytttteeeexxxxtttt....

          iiiinnnntttt YYYYYYYYLLLLeeeennnngggg(((())))
               returns the length of the most recently matched token,
               the equivalent of yyyyyyyylllleeeennnngggg....

          iiiinnnntttt lllliiiinnnneeeennnnoooo(((()))) ccccoooonnnnsssstttt
               returns the current input line number (see %%%%ooooppppttttiiiioooonnnn
               yyyyyyyylllliiiinnnneeeennnnoooo)))),,,, or 1111 if %%%%ooooppppttttiiiioooonnnn yyyyyyyylllliiiinnnneeeennnnoooo was not used.

          vvvvooooiiiidddd sssseeeetttt____ddddeeeebbbbuuuugggg(((( iiiinnnntttt ffffllllaaaagggg ))))
               sets the debugging flag for the scanner, equivalent to
               assigning to yyyyyyyy____fffflllleeeexxxx____ddddeeeebbbbuuuugggg (see the Options section
               above).  Note that you must build the scanner using



     Page 45                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



               %%%%ooooppppttttiiiioooonnnn ddddeeeebbbbuuuugggg to include debugging information in it.

          iiiinnnntttt ddddeeeebbbbuuuugggg(((()))) ccccoooonnnnsssstttt
               returns the current setting of the debugging flag.

          Also provided are member functions equivalent to
          yyyyyyyy____sssswwwwiiiittttcccchhhh____ttttoooo____bbbbuuuuffffffffeeeerrrr(((()))),,,, yyyyyyyy____ccccrrrreeeeaaaatttteeee____bbbbuuuuffffffffeeeerrrr(((()))) (though the first
          argument is an iiiissssttttrrrreeeeaaaammmm**** object pointer and not a FFFFIIIILLLLEEEE****)))),,,,
          yyyyyyyy____fffflllluuuusssshhhh____bbbbuuuuffffffffeeeerrrr(((()))),,,, yyyyyyyy____ddddeeeelllleeeetttteeee____bbbbuuuuffffffffeeeerrrr(((()))),,,, and yyyyyyyyrrrreeeessssttttaaaarrrrtttt(((())))
          (again, the first argument is a iiiissssttttrrrreeeeaaaammmm**** object pointer).

          The second class defined in _F_l_e_x_L_e_x_e_r._h is yyyyyyyyFFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr,,,,
          which is derived from FFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr.... It defines the following
          additional member functions:

          yyyyyyyyFFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr(((( iiiissssttttrrrreeeeaaaammmm**** aaaarrrrgggg____yyyyyyyyiiiinnnn ==== 0000,,,, oooossssttttrrrreeeeaaaammmm**** aaaarrrrgggg____yyyyyyyyoooouuuutttt ==== 0000 ))))
               constructs a yyyyyyyyFFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr object using the given streams
               for input and output.  If not specified, the streams
               default to cccciiiinnnn and ccccoooouuuutttt,,,, respectively.

          vvvviiiirrrrttttuuuuaaaallll iiiinnnntttt yyyyyyyylllleeeexxxx(((())))
               performs the same role is yyyyyyyylllleeeexxxx(((()))) does for ordinary
               flex scanners: it scans the input stream, consuming
               tokens, until a rule's action returns a value.  If you
               derive a subclass SSSS from yyyyyyyyFFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr and want to access
               the member functions and variables of SSSS inside yyyyyyyylllleeeexxxx(((()))),,,,
               then you need to use %%%%ooooppppttttiiiioooonnnn yyyyyyyyccccllllaaaassssssss====""""SSSS"""" to inform _f_l_e_x
               that you will be using that subclass instead of
               yyyyyyyyFFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr.... In this case, rather than generating
               yyyyyyyyFFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr::::::::yyyyyyyylllleeeexxxx(((()))),,,, _f_l_e_x generates SSSS::::::::yyyyyyyylllleeeexxxx(((()))) (and
               also generates a dummy yyyyyyyyFFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr::::::::yyyyyyyylllleeeexxxx(((()))) that calls
               yyyyyyyyFFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr::::::::LLLLeeeexxxxeeeerrrrEEEErrrrrrrroooorrrr(((()))) if called).

          vvvviiiirrrrttttuuuuaaaallll vvvvooooiiiidddd sssswwwwiiiittttcccchhhh____ssssttttrrrreeeeaaaammmmssss((((iiiissssttttrrrreeeeaaaammmm**** nnnneeeewwww____iiiinnnn ==== 0000,,,,
               oooossssttttrrrreeeeaaaammmm**** nnnneeeewwww____oooouuuutttt ==== 0000)))) reassigns yyyyyyyyiiiinnnn to nnnneeeewwww____iiiinnnn (if
               non-nil) and yyyyyyyyoooouuuutttt to nnnneeeewwww____oooouuuutttt (ditto), deleting the
               previous input buffer if yyyyyyyyiiiinnnn is reassigned.

          iiiinnnntttt yyyyyyyylllleeeexxxx(((( iiiissssttttrrrreeeeaaaammmm**** nnnneeeewwww____iiiinnnn,,,, oooossssttttrrrreeeeaaaammmm**** nnnneeeewwww____oooouuuutttt ==== 0000 ))))
               first switches the input streams via sssswwwwiiiittttcccchhhh____ssssttttrrrreeeeaaaammmmssss((((
               nnnneeeewwww____iiiinnnn,,,, nnnneeeewwww____oooouuuutttt )))) and then returns the value of
               yyyyyyyylllleeeexxxx(((())))....

          In addition, yyyyyyyyFFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr defines the following protected
          virtual functions which you can redefine in derived classes
          to tailor the scanner:

          vvvviiiirrrrttttuuuuaaaallll iiiinnnntttt LLLLeeeexxxxeeeerrrrIIIInnnnppppuuuutttt(((( cccchhhhaaaarrrr**** bbbbuuuuffff,,,, iiiinnnntttt mmmmaaaaxxxx____ssssiiiizzzzeeee ))))
               reads up to mmmmaaaaxxxx____ssssiiiizzzzeeee characters into bbbbuuuuffff and returns
               the number of characters read.  To indicate end-of-
               input, return 0 characters.  Note that "interactive"
               scanners (see the ----BBBB and ----IIII flags) define the macro



     Page 46                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



               YYYYYYYY____IIIINNNNTTTTEEEERRRRAAAACCCCTTTTIIIIVVVVEEEE.... If you redefine LLLLeeeexxxxeeeerrrrIIIInnnnppppuuuutttt(((()))) and need
               to take different actions depending on whether or not
               the scanner might be scanning an interactive input
               source, you can test for the presence of this name via
               ####iiiiffffddddeeeeffff....

          vvvviiiirrrrttttuuuuaaaallll vvvvooooiiiidddd LLLLeeeexxxxeeeerrrrOOOOuuuuttttppppuuuutttt(((( ccccoooonnnnsssstttt cccchhhhaaaarrrr**** bbbbuuuuffff,,,, iiiinnnntttt ssssiiiizzzzeeee ))))
               writes out ssssiiiizzzzeeee characters from the buffer bbbbuuuuffff,,,, which,
               while NUL-terminated, may also contain "internal" NUL's
               if the scanner's rules can match text with NUL's in
               them.

          vvvviiiirrrrttttuuuuaaaallll vvvvooooiiiidddd LLLLeeeexxxxeeeerrrrEEEErrrrrrrroooorrrr(((( ccccoooonnnnsssstttt cccchhhhaaaarrrr**** mmmmssssgggg ))))
               reports a fatal error message.  The default version of
               this function writes the message to the stream cccceeeerrrrrrrr and
               exits.

          Note that a yyyyyyyyFFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr object contains its _e_n_t_i_r_e scanning
          state.  Thus you can use such objects to create reentrant
          scanners.  You can instantiate multiple instances of the
          same yyyyyyyyFFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr class, and you can also combine multiple
          C++ scanner classes together in the same program using the
          ----PPPP option discussed above.

          Finally, note that the %%%%aaaarrrrrrrraaaayyyy feature is not available to
          C++ scanner classes; you must use %%%%ppppooooiiiinnnntttteeeerrrr (the default).

          Here is an example of a simple C++ scanner:

                  // An example of using the flex C++ scanner class.

              %{
              int mylineno = 0;
              %}

              string  \"[^\n"]+\"

              ws      [ \t]+

              alpha   [A-Za-z]
              dig     [0-9]
              name    ({alpha}|{dig}|\$)({alpha}|{dig}|[_.\-/$])*
              num1    [-+]?{dig}+\.?([eE][-+]?{dig}+)?
              num2    [-+]?{dig}*\.{dig}+([eE][-+]?{dig}+)?
              number  {num1}|{num2}

              %%

              {ws}    /* skip blanks and tabs */

              "/*"    {
                      int c;



     Page 47                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



                      while((c = yyinput()) != 0)
                          {
                          if(c == '\n')
                              ++mylineno;

                          else if(c == '*')
                              {
                              if((c = yyinput()) == '/')
                                  break;
                              else
                                  unput(c);
                              }
                          }
                      }

              {number}  cout << "number " << YYText() << '\n';

              \n        mylineno++;

              {name}    cout << "name " << YYText() << '\n';

              {string}  cout << "string " << YYText() << '\n';

              %%

              int main( int /* argc */, char** /* argv */ )
                  {
                  FlexLexer* lexer = new yyFlexLexer;
                  while(lexer->yylex() != 0)
                      ;
                  return 0;
                  }
          If you want to create multiple (different) lexer classes,
          you use the ----PPPP flag (or the pppprrrreeeeffffiiiixxxx==== option) to rename each
          yyyyyyyyFFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr to some other xxxxxxxxFFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr.... You then can include
          <<<<FFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr....hhhh>>>> in your other sources once per lexer class,
          first renaming yyyyyyyyFFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr as follows:

              #undef yyFlexLexer
              #define yyFlexLexer xxFlexLexer
              #include <FlexLexer.h>

              #undef yyFlexLexer
              #define yyFlexLexer zzFlexLexer
              #include <FlexLexer.h>

          if, for example, you used %%%%ooooppppttttiiiioooonnnn pppprrrreeeeffffiiiixxxx====""""xxxxxxxx"""" for one of
          your scanners and %%%%ooooppppttttiiiioooonnnn pppprrrreeeeffffiiiixxxx====""""zzzzzzzz"""" for the other.

          IMPORTANT: the present form of the scanning class is
          _e_x_p_e_r_i_m_e_n_t_a_l and may change considerably between major
          releases.



     Page 48                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



     IIIINNNNCCCCOOOOMMMMPPPPAAAATTTTIIIIBBBBIIIILLLLIIIITTTTIIIIEEEESSSS WWWWIIIITTTTHHHH LLLLEEEEXXXX AAAANNNNDDDD PPPPOOOOSSSSIIIIXXXX
          _f_l_e_x is a rewrite of the AT&T Unix _l_e_x tool (the two
          implementations do not share any code, though), with some
          extensions and incompatibilities, both of which are of
          concern to those who wish to write scanners acceptable to
          either implementation.  Flex is fully compliant with the
          POSIX _l_e_x specification, except that when using %%%%ppppooooiiiinnnntttteeeerrrr
          (the default), a call to uuuunnnnppppuuuutttt(((()))) destroys the contents of
          yyyyyyyytttteeeexxxxtttt,,,, which is counter to the POSIX specification.

          In this section we discuss all of the known areas of
          incompatibility between flex, AT&T lex, and the POSIX
          specification.

          _f_l_e_x'_s ----llll option turns on maximum compatibility with the
          original AT&T _l_e_x implementation, at the cost of a major
          loss in the generated scanner's performance.  We note below
          which incompatibilities can be overcome using the ----llll option.

          _f_l_e_x is fully compatible with _l_e_x with the following
          exceptions:

          -    The undocumented _l_e_x scanner internal variable yyyyyyyylllliiiinnnneeeennnnoooo
               is not supported unless ----llll or %%%%ooooppppttttiiiioooonnnn yyyyyyyylllliiiinnnneeeennnnoooo is used.

               yyyyyyyylllliiiinnnneeeennnnoooo should be maintained on a per-buffer basis,
               rather than a per-scanner (single global variable)
               basis.

               yyyyyyyylllliiiinnnneeeennnnoooo is not part of the POSIX specification.

          -    The iiiinnnnppppuuuutttt(((()))) routine is not redefinable, though it may
               be called to read characters following whatever has
               been matched by a rule.  If iiiinnnnppppuuuutttt(((()))) encounters an end-
               of-file the normal yyyyyyyywwwwrrrraaaapppp(((()))) processing is done.  A
               ``real'' end-of-file is returned by iiiinnnnppppuuuutttt(((()))) as _E_O_F.

               Input is instead controlled by defining the YYYYYYYY____IIIINNNNPPPPUUUUTTTT
               macro.

               The _f_l_e_x restriction that iiiinnnnppppuuuutttt(((()))) cannot be redefined
               is in accordance with the POSIX specification, which
               simply does not specify any way of controlling the
               scanner's input other than by making an initial
               assignment to _y_y_i_n.

          -    The uuuunnnnppppuuuutttt(((()))) routine is not redefinable.  This
               restriction is in accordance with POSIX.

          -    _f_l_e_x scanners are not as reentrant as _l_e_x scanners.  In
               particular, if you have an interactive scanner and an
               interrupt handler which long-jumps out of the scanner,



     Page 49                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



               and the scanner is subsequently called again, you may
               get the following message:

                   fatal flex scanner internal error--end of buffer missed

               To reenter the scanner, first use

                   yyrestart( yyin );

               Note that this call will throw away any buffered input;
               usually this isn't a problem with an interactive
               scanner.

               Also note that flex C++ scanner classes _a_r_e reentrant,
               so if using C++ is an option for you, you should use
               them instead.  See "Generating C++ Scanners" above for
               details.

          -    oooouuuuttttppppuuuutttt(((()))) is not supported.  Output from the EEEECCCCHHHHOOOO macro
               is done to the file-pointer _y_y_o_u_t (default _s_t_d_o_u_t).

               oooouuuuttttppppuuuutttt(((()))) is not part of the POSIX specification.

          -    _l_e_x does not support exclusive start conditions (%x),
               though they are in the POSIX specification.

          -    When definitions are expanded, _f_l_e_x encloses them in
               parentheses.  With lex, the following:

                   NAME    [A-Z][A-Z0-9]*
                   %%
                   foo{NAME}?      printf( "Found it\n" );
                   %%

               will not match the string "foo" because when the macro
               is expanded the rule is equivalent to "foo[A-Z][A-Z0-
               9]*?"  and the precedence is such that the '?' is
               associated with "[A-Z0-9]*".  With _f_l_e_x, the rule will
               be expanded to "foo([A-Z][A-Z0-9]*)?" and so the string
               "foo" will match.

               Note that if the definition begins with ^^^^ or ends with
               $$$$ then it is _n_o_t expanded with parentheses, to allow
               these operators to appear in definitions without losing
               their special meanings.  But the <<<<ssss>>>>,,,, ////,,,, and <<<<<<<<EEEEOOOOFFFF>>>>>>>>
               operators cannot be used in a _f_l_e_x definition.

               Using ----llll results in the _l_e_x behavior of no parentheses
               around the definition.

               The POSIX specification is that the definition be
               enclosed in parentheses.



     Page 50                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          -    Some implementations of _l_e_x allow a rule's action to
               begin on a separate line, if the rule's pattern has
               trailing whitespace:

                   %%
                   foo|bar<space here>
                     { foobar_action(); }

               _f_l_e_x does not support this feature.

          -    The _l_e_x %%%%rrrr (generate a Ratfor scanner) option is not
               supported.  It is not part of the POSIX specification.

          -    After a call to uuuunnnnppppuuuutttt(((()))),,,, _y_y_t_e_x_t is undefined until the
               next token is matched, unless the scanner was built
               using %%%%aaaarrrrrrrraaaayyyy.... This is not the case with _l_e_x or the
               POSIX specification.  The ----llll option does away with this
               incompatibility.

          -    The precedence of the {{{{}}}} (numeric range) operator is
               different.  The AT&T and POSIX specifiations of _l_e_x
               interpret "abc{1,3}" as "match one, two, or three
               occurrences of 'abc'", whereas _f_l_e_x interprets it as
               "match 'ab' followed by one, two, or three occurrences
               of 'c'".  The ----llll and --------ppppoooossssiiiixxxx----ccccoooommmmppppaaaatttt options do away
               with this incompatibility.

          -    The precedence of the ^^^^ operator is different.  _l_e_x
               interprets "^foo|bar" as "match either 'foo' at the
               beginning of a line, or 'bar' anywhere", whereas _f_l_e_x
               interprets it as "match either 'foo' or 'bar' if they
               come at the beginning of a line".  The latter is in
               agreement with the POSIX specification.

          -    The special table-size declarations such as %%%%aaaa
               supported by _l_e_x are not required by _f_l_e_x scanners;
               _f_l_e_x ignores them.

          -    The name FLEX_SCANNER is #define'd so scanners may be
               written for use with either _f_l_e_x or _l_e_x. Scanners also
               include YYYYYYYY____FFFFLLLLEEEEXXXX____MMMMAAAAJJJJOOOORRRR____VVVVEEEERRRRSSSSIIIIOOOONNNN and YYYYYYYY____FFFFLLLLEEEEXXXX____MMMMIIIINNNNOOOORRRR____VVVVEEEERRRRSSSSIIIIOOOONNNN
               indicating which version of _f_l_e_x generated the scanner
               (for example, for the 2.5 release, these defines would
               be 2 and 5 respectively).

          The following _f_l_e_x features are not included in _l_e_x or the
          POSIX specification:

              C++ scanners
              %option
              start condition scopes
              start condition stacks



     Page 51                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



              interactive/non-interactive scanners
              yy_scan_string() and friends
              yyterminate()
              yy_set_interactive()
              yy_set_bol()
              YY_AT_BOL()
              <<EOF>>
              <*>
              YY_DECL
              YY_START
              YY_USER_ACTION
              YY_USER_INIT
              #line directives
              %{}'s around actions
              multiple actions on a line

          plus almost all of the flex flags.  The last feature in the
          list refers to the fact that with _f_l_e_x you can put multiple
          actions on the same line, separated with semi-colons, while
          with _l_e_x, the following

              foo    handle_foo(); ++num_foos_seen;

          is (rather surprisingly) truncated to

              foo    handle_foo();

          _f_l_e_x does not truncate the action.  Actions that are not
          enclosed in braces are simply terminated at the end of the
          line.

     DDDDIIIIAAAAGGGGNNNNOOOOSSSSTTTTIIIICCCCSSSS
          _w_a_r_n_i_n_g, _r_u_l_e _c_a_n_n_o_t _b_e _m_a_t_c_h_e_d indicates that the given
          rule cannot be matched because it follows other rules that
          will always match the same text as it.  For example, in the
          following "foo" cannot be matched because it comes after an
          identifier "catch-all" rule:

              [a-z]+    got_identifier();
              foo       got_foo();

          Using RRRREEEEJJJJEEEECCCCTTTT in a scanner suppresses this warning.

          _w_a_r_n_i_n_g, ----ssss _o_p_t_i_o_n _g_i_v_e_n _b_u_t _d_e_f_a_u_l_t _r_u_l_e _c_a_n _b_e _m_a_t_c_h_e_d
          means that it is possible (perhaps only in a particular
          start condition) that the default rule (match any single
          character) is the only one that will match a particular
          input.  Since ----ssss was given, presumably this is not intended.

          _r_e_j_e_c_t__u_s_e_d__b_u_t__n_o_t__d_e_t_e_c_t_e_d _u_n_d_e_f_i_n_e_d or
          _y_y_m_o_r_e__u_s_e_d__b_u_t__n_o_t__d_e_t_e_c_t_e_d _u_n_d_e_f_i_n_e_d - These errors can
          occur at compile time.  They indicate that the scanner uses



     Page 52                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          RRRREEEEJJJJEEEECCCCTTTT or yyyyyyyymmmmoooorrrreeee(((()))) but that _f_l_e_x failed to notice the fact,
          meaning that _f_l_e_x scanned the first two sections looking for
          occurrences of these actions and failed to find any, but
          somehow you snuck some in (via a #include file, for
          example).  Use %%%%ooooppppttttiiiioooonnnn rrrreeeejjjjeeeecccctttt or %%%%ooooppppttttiiiioooonnnn yyyyyyyymmmmoooorrrreeee to indicate
          to flex that you really do use these features.

          _f_l_e_x _s_c_a_n_n_e_r _j_a_m_m_e_d - a scanner compiled with ----ssss has
          encountered an input string which wasn't matched by any of
          its rules.  This error can also occur due to internal
          problems.

          _t_o_k_e_n _t_o_o _l_a_r_g_e, _e_x_c_e_e_d_s _Y_Y_L_M_A_X - your scanner uses %%%%aaaarrrrrrrraaaayyyy
          and one of its rules matched a string longer than the YYYYYYYYLLLLMMMMAAAAXXXX
          constant (8K bytes by default).  You can increase the value
          by #define'ing YYYYYYYYLLLLMMMMAAAAXXXX in the definitions section of your
          _f_l_e_x input.

          _s_c_a_n_n_e_r _r_e_q_u_i_r_e_s -_8 _f_l_a_g _t_o _u_s_e _t_h_e _c_h_a_r_a_c_t_e_r '_x' - Your
          scanner specification includes recognizing the 8-bit
          character '_x' and you did not specify the -8 flag, and your
          scanner defaulted to 7-bit because you used the ----CCCCffff or ----CCCCFFFF
          table compression options.  See the discussion of the ----7777
          flag for details.

          _f_l_e_x _s_c_a_n_n_e_r _p_u_s_h-_b_a_c_k _o_v_e_r_f_l_o_w - you used uuuunnnnppppuuuutttt(((()))) to push
          back so much text that the scanner's buffer could not hold
          both the pushed-back text and the current token in yyyyyyyytttteeeexxxxtttt....
          Ideally the scanner should dynamically resize the buffer in
          this case, but at present it does not.

          _i_n_p_u_t _b_u_f_f_e_r _o_v_e_r_f_l_o_w, _c_a_n'_t _e_n_l_a_r_g_e _b_u_f_f_e_r _b_e_c_a_u_s_e _s_c_a_n_n_e_r
          _u_s_e_s _R_E_J_E_C_T - the scanner was working on matching an
          extremely large token and needed to expand the input buffer.
          This doesn't work with scanners that use RRRREEEEJJJJEEEECCCCTTTT....

          _f_a_t_a_l _f_l_e_x _s_c_a_n_n_e_r _i_n_t_e_r_n_a_l _e_r_r_o_r--_e_n_d _o_f _b_u_f_f_e_r _m_i_s_s_e_d -
          This can occur in an scanner which is reentered after a
          long-jump has jumped out (or over) the scanner's activation
          frame.  Before reentering the scanner, use:

              yyrestart( yyin );

          or, as noted above, switch to using the C++ scanner class.

          _t_o_o _m_a_n_y _s_t_a_r_t _c_o_n_d_i_t_i_o_n_s _i_n <> you listed more start
          conditions in a <> construct than exist (so you must have
          listed at least one of them twice).

     FFFFIIIILLLLEEEESSSS
          ----llllffffllll library with which scanners must be linked.




     Page 53                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          _l_e_x._y_y._c
               generated scanner (called _l_e_x_y_y._c on some systems).

          _l_e_x._y_y._c_c
               generated C++ scanner class, when using ----++++....

          <_F_l_e_x_L_e_x_e_r._h>
               header file defining the C++ scanner base class,
               FFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr,,,, and its derived class, yyyyyyyyFFFFlllleeeexxxxLLLLeeeexxxxeeeerrrr....

          _f_l_e_x._s_k_l
               skeleton scanner.  This file is only used when building
               flex, not when flex executes.

          _l_e_x._b_a_c_k_u_p
               backing-up information for ----bbbb flag (called _l_e_x._b_c_k on
               some systems).

     DDDDEEEEFFFFIIIICCCCIIIIEEEENNNNCCCCIIIIEEEESSSS //// BBBBUUUUGGGGSSSS
          Some trailing context patterns cannot be properly matched
          and generate warning messages ("dangerous trailing
          context").  These are patterns where the ending of the first
          part of the rule matches the beginning of the second part,
          such as "zx*/xy*", where the 'x*' matches the 'x' at the
          beginning of the trailing context.  (Note that the POSIX
          draft states that the text matched by such patterns is
          undefined.)

          For some trailing context rules, parts which are actually
          fixed-length are not recognized as such, leading to the
          abovementioned performance loss.  In particular, parts using
          '|' or {n} (such as "foo{3}") are always considered
          variable-length.

          Combining trailing context with the special '|' action can
          result in _f_i_x_e_d trailing context being turned into the more
          expensive _v_a_r_i_a_b_l_e trailing context.  For example, in the
          following:

              %%
              abc      |
              xyz/def


          Use of uuuunnnnppppuuuutttt(((()))) invalidates yytext and yyleng, unless the
          %%%%aaaarrrrrrrraaaayyyy directive or the ----llll option has been used.

          Pattern-matching of NUL's is substantially slower than
          matching other characters.

          Dynamic resizing of the input buffer is slow, as it entails
          rescanning all the text matched so far by the current



     Page 54                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          (generally huge) token.

          Due to both buffering of input and read-ahead, you cannot
          intermix calls to <stdio.h> routines, such as, for example,
          ggggeeeettttcccchhhhaaaarrrr(((()))),,,, with _f_l_e_x rules and expect it to work.  Call
          iiiinnnnppppuuuutttt(((()))) instead.

          The total table entries listed by the ----vvvv flag excludes the
          number of table entries needed to determine what rule has
          been matched.  The number of entries is equal to the number
          of DFA states if the scanner does not use RRRREEEEJJJJEEEECCCCTTTT,,,, and
          somewhat greater than the number of states if it does.

          RRRREEEEJJJJEEEECCCCTTTT cannot be used with the ----ffff or ----FFFF options.

          The _f_l_e_x internal algorithms need documentation.

     SSSSEEEEEEEE AAAALLLLSSSSOOOO
          lex(1), yacc(1), sed(1), awk(1).

          John Levine, Tony Mason, and Doug Brown, _L_e_x & _Y_a_c_c,
          O'Reilly and Associates.  Be sure to get the 2nd edition.

          M. E. Lesk and E. Schmidt, _L_E_X - _L_e_x_i_c_a_l _A_n_a_l_y_z_e_r _G_e_n_e_r_a_t_o_r

          Alfred Aho, Ravi Sethi and Jeffrey Ullman, _C_o_m_p_i_l_e_r_s:
          _P_r_i_n_c_i_p_l_e_s, _T_e_c_h_n_i_q_u_e_s _a_n_d _T_o_o_l_s, Addison-Wesley (1986).
          Describes the pattern-matching techniques used by _f_l_e_x
          (deterministic finite automata).

     AAAAUUUUTTTTHHHHOOOORRRR
          Vern Paxson, with the help of many ideas and much
          inspiration from Van Jacobson.  Original version by Jef
          Poskanzer.  The fast table representation is a partial
          implementation of a design done by Van Jacobson.  The
          implementation was done by Kevin Gong and Vern Paxson.

          Thanks to the many _f_l_e_x beta-testers, feedbackers, and
          contributors, especially Francois Pinard, Casey Leedom,
          Robert Abramovitz, Stan Adermann, Terry Allen, David
          Barker-Plummer, John Basrai, Neal Becker, Nelson H.F. Beebe,
          [e-mail], Karl Berry, Peter A. Bigot, Simon Blanchard,
          Keith Bostic, Frederic Brehm, Ian Brockbank, Kin Cho, Nick
          Christopher, Brian Clapper, J.T. Conklin, Jason Coughlin,
          Bill Cox, Nick Cropper, Dave Curtis, Scott David Daniels,
          Chris G. Demetriou, Theo Deraadt, Mike Donahue, Chuck
          Doucette, Tom Epperly, Leo Eskin, Chris Faylor, Chris
          Flatters, Jon Forrest, Jeffrey Friedl, Joe Gayda, Kaveh R.
          Ghazi, Wolfgang Glunz, Eric Goldman, Christopher M. Gould,
          Ulrich Grepel, Peer Griebel, Jan Hajic, Charles Hemphill,
          NORO Hideo, Jarkko Hietaniemi, Scott Hofmann, Jeff Honig,
          Dana Hudes, Eric Hughes, John Interrante, Ceriel Jacobs,



     Page 55                                         (printed 10/3/02)






     FFFFLLLLEEEEXXXX((((1111))))             VVVVeeeerrrrssssiiiioooonnnn 2222....5555 ((((AAAApppprrrriiiillll 1111999999995555))))              FFFFLLLLEEEEXXXX((((1111))))



          Michal Jaegermann, Sakari Jalovaara, Jeffrey R. Jones, Henry
          Juengst, Klaus Kaempf, Jonathan I. Kamens, Terrence O Kane,
          Amir Katz, [e-mail], Kevin B. Kenny, Steve Kirsch,
          Winfried Koenig, Marq Kole, Ronald Lamprecht, Greg Lee,
          Rohan Lenard, Craig Leres, John Levine, Steve Liddle, David
          Loffredo, Mike Long, Mohamed el Lozy, Brian Madsen, Malte,
          Joe Marshall, Bengt Martensson, Chris Metcalf, Luke Mewburn,
          Jim Meyering, R. Alexander Milowski, Erik Naggum, G.T.
          Nicol, Landon Noll, James Nordby, Marc Nozell, Richard
          Ohnemus, Karsten Pahnke, Sven Panne, Roland Pesch, Walter
          Pelissero, Gaumond Pierre, Esmond Pitt, Jef Poskanzer, Joe
          Rahmeh, Jarmo Raiha, Frederic Raimbault, Pat Rankin, Rick
          Richardson, Kevin Rodgers, Kai Uwe Rommel, Jim Roskind,
          Alberto Santini, Andreas Scherer, Darrell Schiebel, Raf
          Schietekat, Doug Schmidt, Philippe Schnoebelen, Andreas
          Schwab, Larry Schwimmer, Alex Siegel, Eckehard Stolz, Jan-
          Erik Strvmquist, Mike Stump, Paul Stuart, Dave Tallman, Ian
          Lance Taylor, Chris Thewalt, Richard M. Timoney, Jodi Tsai,
          Paul Tuinenga, Gary Weik, Frank Whaley, Gerhard Wilhelms,
          Kent Williams, Ken Yap, Ron Zellar, Nathan Zelle, David
          Zuhn, and those whose names have slipped my marginal mail-
          archiving skills but whose contributions are appreciated all
          the same.

          Thanks to Keith Bostic, Jon Forrest, Noah Friedman, John
          Gilmore, Craig Leres, John Levine, Bob Mulcahy, G.T.  Nicol,
          Francois Pinard, Rich Salz, and Richard Stallman for help
          with various distribution headaches.

          Thanks to Esmond Pitt and Earle Horton for 8-bit character
          support; to Benson Margulies and Fred Burke for C++ support;
          to Kent Williams and Tom Epperly for C++ class support; to
          Ove Ewerlid for support of NUL's; and to Eric Hughes for
          support of multiple buffers.

          This work was primarily done when I was with the Real Time
          Systems Group at the Lawrence Berkeley Laboratory in
          Berkeley, CA.  Many thanks to all there for the support I
          received.

          Send comments to [e-mail].














     Page 56                                         (printed 10/3/02)